[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:29 dist-test-slave-xd2q krb5kdc[19725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:29 dist-test-slave-xd2q krb5kdc[19725](info): set up 2 sockets
Jun 19 10:21:29 dist-test-slave-xd2q krb5kdc[19725](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 10:21:30.587968 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:30 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:30.704282 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41877
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:30.782584 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:30.782810 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:30.782850 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:30.783986 19738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:30.784055 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:30.784080 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:30.784101 19738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:30.784130 19738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:30.785548 19738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:30.785818 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:30.786049 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:30.791074 19744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:30.791041 19746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:30.791046 19743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:30.791309 19738 server_base.cc:1034] running on GCE node
I20260619 10:21:30.791548 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:30.791838 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:30.793015 19738 hybrid_clock.cc:648] HybridClock initialized: now 1781864490793003 us; error 27 us; skew 500 ppm
Jun 19 10:21:30 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:30.795481 19738 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:30.795892 19738 webserver.cc:460] Webserver started at http://127.19.65.190:33001/ using document root <none> and password file <none>
I20260619 10:21:30.796113 19738 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:30.796156 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:30.796303 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:30.797346 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "5da28020205040a58edd4cb02680ecd2"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "b474e5d7d83a059dc7f92b73dca8fb6e"
server_key_iv: "6f08446e27a0b37ce4b8bcbed18d3b73"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.797712 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "5da28020205040a58edd4cb02680ecd2"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "b474e5d7d83a059dc7f92b73dca8fb6e"
server_key_iv: "6f08446e27a0b37ce4b8bcbed18d3b73"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.799075 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:30.799902 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:30.800077 19738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:30.800173 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "5da28020205040a58edd4cb02680ecd2"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "b474e5d7d83a059dc7f92b73dca8fb6e"
server_key_iv: "6f08446e27a0b37ce4b8bcbed18d3b73"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.800258 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:30.812094 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:30.813764 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:30.813920 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:30.817956 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:41877
I20260619 10:21:30.817983 19805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:41877 every 8 connection(s)
I20260619 10:21:30.818351 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:30.818805 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 19738
I20260619 10:21:30.818918 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:30.819375 19806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:30.819729 19718 external_mini_cluster.cc:1428] Setting key 9e5ecffdf2102fb7edd30159f682d144
I20260619 10:21:30.822328 19806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2: Bootstrap starting.
I20260619 10:21:30.822968 19806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:30.823338 19806 log.cc:824] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:30.824049 19806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2: No bootstrap required, opened a new log
I20260619 10:21:30.826436 19806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } }
I20260619 10:21:30.826556 19806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:30.826577 19806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da28020205040a58edd4cb02680ecd2, State: Initialized, Role: FOLLOWER
I20260619 10:21:30.826704 19806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } }
I20260619 10:21:30.826799 19806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:30.826821 19806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:30.826890 19806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:30.827438 19806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } }
I20260619 10:21:30.827553 19806 leader_election.cc:302] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5da28020205040a58edd4cb02680ecd2; no voters:
I20260619 10:21:30.827739 19806 leader_election.cc:288] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:30.827765 19811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:30.827890 19811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [term 1 LEADER]: Becoming Leader. State: Replica: 5da28020205040a58edd4cb02680ecd2, State: Running, Role: LEADER
I20260619 10:21:30.827973 19811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } }
I20260619 10:21:30.827987 19806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:30.828346 19811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5da28020205040a58edd4cb02680ecd2. Latest consensus state: current_term: 1 leader_uuid: "5da28020205040a58edd4cb02680ecd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } } }
I20260619 10:21:30.828413 19811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:30.828575 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5da28020205040a58edd4cb02680ecd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da28020205040a58edd4cb02680ecd2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41877 } } }
I20260619 10:21:30.828699 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:30.829219 19821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:30.829658 19821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:30.831202 19821 catalog_manager.cc:1347] Generated new cluster ID: 0d247b7deeda47cf857d806e7973f5c8
I20260619 10:21:30.831257 19821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:30 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:30.840451 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:30.820608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.19.65.190:41877)
0619 10:21:30.820760 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:30.820768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:30.820982 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:21:30.823694 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:30.823700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:30.823742 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:30.823847 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:30.825857 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:30.826811 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:30.828117 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:30.828652 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:30.828732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:30.833535 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:30.833540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:30.833540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:30.833552 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:30.837137 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:30.837911 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:30.837914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:30.837914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:30.838086 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:30.838521 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:30.838522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:30.838522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:30.838959 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:30.839166 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:30.839885 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 10:21:30.840125 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260619 10:21:30.850917 19821 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:30.851454 19821 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:30.857647 19821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5da28020205040a58edd4cb02680ecd2: Generated new TSK 0
I20260619 10:21:30.857805 19821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:30.884030 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:30.982678 19832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:30.982861 19832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:30.982882 19832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:30.984037 19832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:30.984076 19832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:30.984100 19832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:30.985543 19832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:30.985762 19832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:30.986007 19832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:30.990697 19838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:30.990741 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:30.990759 19832 server_base.cc:1034] running on GCE node
W20260619 10:21:30.990695 19837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:30.991034 19832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:30.991217 19832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:30.992344 19832 hybrid_clock.cc:648] HybridClock initialized: now 1781864490992331 us; error 26 us; skew 500 ppm
Jun 19 10:21:30 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:30.994629 19832 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:30.995051 19832 webserver.cc:460] Webserver started at http://127.19.65.129:38727/ using document root <none> and password file <none>
I20260619 10:21:30.995229 19832 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:30.995267 19832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:30.995414 19832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:30.996359 19832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "422a8ba6733c4e509eed1ea78efa905c"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "1e36de3e19ed2b84779b577ed78963c6"
server_key_iv: "3154fcc10ae6fb0dbe0b53132c87ed67"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.996642 19832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "422a8ba6733c4e509eed1ea78efa905c"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "1e36de3e19ed2b84779b577ed78963c6"
server_key_iv: "3154fcc10ae6fb0dbe0b53132c87ed67"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.997753 19832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:30.998364 19847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:30.998534 19832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:30.998626 19832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "422a8ba6733c4e509eed1ea78efa905c"
format_stamp: "Formatted at 2026-06-19 10:21:30 on dist-test-slave-xd2q"
server_key: "1e36de3e19ed2b84779b577ed78963c6"
server_key_iv: "3154fcc10ae6fb0dbe0b53132c87ed67"
server_key_version: "encryptionkey@0"
I20260619 10:21:30.998700 19832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:31.012539 19832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:31.014154 19832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:31.014304 19832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:31.014529 19832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:31.014811 19832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:31.014864 19832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.014904 19832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:31.014931 19832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.020735 19832 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:36411
I20260619 10:21:31.020784 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:36411 every 8 connection(s)
I20260619 10:21:31.021059 19832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864490, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:31.028621 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 19832
I20260619 10:21:31.028715 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:31.028874 19718 external_mini_cluster.cc:1428] Setting key 341cf41433c701ae5db17d54fda349ec
I20260619 10:21:31.029842 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.021661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:54967 (local address 127.19.65.190:41877)
0619 10:21:31.021730 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:31.021732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:31.021901 (+ 169us) server_negotiation.cc:407] Connection header received
0619 10:21:31.022334 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:31.022335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:31.022344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:31.022386 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:31.023161 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.023707 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.024636 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.024823 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.024870 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.026821 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:31.026823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:31.026823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:31.026829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:31.028258 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.028806 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.028807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.028808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.028871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.029126 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.029127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.029127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.029268 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:31.029325 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:31.029614 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 10:21:31.029728 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:21:31.030038 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.021634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41877 (local address 127.19.65.129:54967)
0619 10:21:31.021793 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:31.021824 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:31.022246 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:31.022440 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:31.022443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:31.022927 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:31.023097 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.023101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.023777 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.023778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.024563 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.024565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.024888 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.024889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.024981 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.025384 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:31.025391 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:31.026741 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:31.028321 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.028326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.028327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.028672 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.028949 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.028950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.028950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.029054 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.029330 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:31.029335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:31.029470 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 10:21:31.029582 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 10:21:31.029760 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260619 10:21:31.030470 19961 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41877
I20260619 10:21:31.030573 19961 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:31.030728 19961 heartbeater.cc:511] Master 127.19.65.190:41877 requested a full tablet report, sending...
I20260619 10:21:31.031306 19770 ts_manager.cc:194] Registered new tserver with Master: 422a8ba6733c4e509eed1ea78efa905c (127.19.65.129:36411)
I20260619 10:21:31.031940 19770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:54967
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:31.053707 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:31.130501 19966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:31.130712 19966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:31.130749 19966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:31.131846 19966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:31.131909 19966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:31.131953 19966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:31.133406 19966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:31.133671 19966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:31.133891 19966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:31.138672 19972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:31.138680 19974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:31.138702 19971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:31.138963 19966 server_base.cc:1034] running on GCE node
I20260619 10:21:31.139153 19966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:31.139425 19966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:31.140591 19966 hybrid_clock.cc:648] HybridClock initialized: now 1781864491140573 us; error 44 us; skew 500 ppm
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:31.143007 19966 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:31.143472 19966 webserver.cc:460] Webserver started at http://127.19.65.130:40171/ using document root <none> and password file <none>
I20260619 10:21:31.143703 19966 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:31.143766 19966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:31.143896 19966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:31.144906 19966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "73c98d35fc9b4cfbbcc1657c1c05de8e"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "1bd1d91025a8d68efc5883cefffb232f"
server_key_iv: "ec8193c541ef93953551cd9e13bceb41"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.145272 19966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "73c98d35fc9b4cfbbcc1657c1c05de8e"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "1bd1d91025a8d68efc5883cefffb232f"
server_key_iv: "ec8193c541ef93953551cd9e13bceb41"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.146541 19966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:31.147219 19981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.147383 19966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:31.147485 19966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "73c98d35fc9b4cfbbcc1657c1c05de8e"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "1bd1d91025a8d68efc5883cefffb232f"
server_key_iv: "ec8193c541ef93953551cd9e13bceb41"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.147567 19966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:31.155550 19966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:31.157771 19966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:31.157928 19966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:31.158128 19966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:31.158416 19966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:31.158488 19966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.158526 19966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:31.158553 19966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.164296 19966 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45825
I20260619 10:21:31.164332 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:45825 every 8 connection(s)
I20260619 10:21:31.164652 19966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:31.167539 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 19966
I20260619 10:21:31.167631 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:31.167805 19718 external_mini_cluster.cc:1428] Setting key 31fbf33a0f82fca4d672a9e4d5d10905
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:31.174487 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.165337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:35203 (local address 127.19.65.190:41877)
0619 10:21:31.165397 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:31.165399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:31.165580 (+ 181us) server_negotiation.cc:407] Connection header received
0619 10:21:31.166036 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:31.166037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:31.166046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:31.166100 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:31.166920 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.167523 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.168820 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.169137 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.169214 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.171713 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:31.171714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:31.171715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:31.171722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:31.173134 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.173552 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.173553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.173553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.173612 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.173853 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.173854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.173854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.173972 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:31.174050 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:31.174314 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 10:21:31.174394 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:21:31.174693 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.165318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41877 (local address 127.19.65.130:35203)
0619 10:21:31.165458 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:31.165496 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:31.165955 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:31.166149 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:31.166152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:31.166672 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:31.166847 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.166851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.167629 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.167630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.168728 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.168730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.169245 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.169246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.169522 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.170054 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:31.170066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:31.171618 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:31.173233 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.173239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.173241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.173461 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.173678 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.173679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.173679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.173786 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.174042 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:31.174048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:31.174185 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 10:21:31.174293 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 10:21:31.174455 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260619 10:21:31.175138 20095 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41877
I20260619 10:21:31.175228 20095 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:31.175432 20095 heartbeater.cc:511] Master 127.19.65.190:41877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:21:31.175822 19770 ts_manager.cc:194] Registered new tserver with Master: 73c98d35fc9b4cfbbcc1657c1c05de8e (127.19.65.130:45825)
I20260619 10:21:31.176256 19770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:35203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:31.193511 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:31.272768 20100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:31.272994 20100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:31.273031 20100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:31.274204 20100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:31.274271 20100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:31.274319 20100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:31.275791 20100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:31.276054 20100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:31.276268 20100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:31.281358 20108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:31.281411 20106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:31.281405 20105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:31.281701 20100 server_base.cc:1034] running on GCE node
I20260619 10:21:31.281925 20100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:31.282166 20100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:31.283370 20100 hybrid_clock.cc:648] HybridClock initialized: now 1781864491283343 us; error 38 us; skew 500 ppm
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:31.285773 20100 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:31.286237 20100 webserver.cc:460] Webserver started at http://127.19.65.131:35147/ using document root <none> and password file <none>
I20260619 10:21:31.286458 20100 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:31.286504 20100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:31.286648 20100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:31.287752 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "34b8b76630f349869dd147a9b6f601e7"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "9f81fa24a121621bf9318e929e4cb797"
server_key_iv: "1e36822ed0bc9249a1639189c3d615ca"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.288110 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "34b8b76630f349869dd147a9b6f601e7"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "9f81fa24a121621bf9318e929e4cb797"
server_key_iv: "1e36822ed0bc9249a1639189c3d615ca"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.289372 20100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:31.290041 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.290217 20100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:31.290310 20100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "34b8b76630f349869dd147a9b6f601e7"
format_stamp: "Formatted at 2026-06-19 10:21:31 on dist-test-slave-xd2q"
server_key: "9f81fa24a121621bf9318e929e4cb797"
server_key_iv: "1e36822ed0bc9249a1639189c3d615ca"
server_key_version: "encryptionkey@0"
I20260619 10:21:31.290422 20100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:31.308945 20100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:31.310626 20100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:31.310796 20100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:31.311008 20100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:31.311352 20100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:31.311409 20100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.311446 20100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:31.311476 20100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:31.317379 20100 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:44135
I20260619 10:21:31.317431 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:44135 every 8 connection(s)
I20260619 10:21:31.317708 20100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:31.318133 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20100
I20260619 10:21:31.318229 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:31.318429 19718 external_mini_cluster.cc:1428] Setting key b5abd00e8b0b4831d31ba4b8b4669dbd
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:31.326522 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.318341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:44991 (local address 127.19.65.190:41877)
0619 10:21:31.318385 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:31.318387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:31.318677 (+ 290us) server_negotiation.cc:407] Connection header received
0619 10:21:31.319303 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:31.319305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:31.319317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:31.319357 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:31.320393 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.320886 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.321910 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.322125 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.322211 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.324229 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:31.324230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:31.324230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:31.324237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:31.325423 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.325786 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.325787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.325787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.325839 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.326042 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.326043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.326043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.326137 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:31.326186 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:31.326399 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 10:21:31.326456 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:31.326683 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.318354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41877 (local address 127.19.65.131:44991)
0619 10:21:31.318545 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:31.318589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:31.319184 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:31.319435 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:31.319439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:31.320096 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:31.320278 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.320283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.320992 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.320993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.321817 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:31.321819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.322218 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.322219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:31.322351 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.322853 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:31.322861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:31.324148 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:31.325499 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.325505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.325506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.325716 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.325894 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:31.325895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:31.325895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:31.325988 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:31.326192 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:31.326196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:31.326297 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 10:21:31.326374 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:21:31.326491 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260619 10:21:31.327112 20229 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41877
I20260619 10:21:31.327220 20229 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:31.327416 20229 heartbeater.cc:511] Master 127.19.65.190:41877 requested a full tablet report, sending...
I20260619 10:21:31.327792 19770 ts_manager.cc:194] Registered new tserver with Master: 34b8b76630f349869dd147a9b6f601e7 (127.19.65.131:44135)
I20260619 10:21:31.328182 19770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:44991
I20260619 10:21:31.330356 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:31.351713 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.344661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.19.65.129:36411)
0619 10:21:31.344833 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:31.344838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:31.344848 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:31.344872 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:31.344874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:31.344887 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:31.344958 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:31.345414 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.346112 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.346750 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.347045 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.347094 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.349083 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:31.349089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:31.349089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:31.349095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:31.350642 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.350927 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.350932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.350933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.351056 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.351207 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.351207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.351207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.351412 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:31.351526 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:31.351581 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 10:21:31.351641 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
W20260619 10:21:31.352070 19875 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41416
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:31.372187 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:31.366096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.19.65.129:36411)
0619 10:21:31.366157 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:31.366159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:31.366216 (+ 57us) server_negotiation.cc:407] Connection header received
0619 10:21:31.366257 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:31.366258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:31.366268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:31.366315 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:31.366789 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.367236 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.367958 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:31.368153 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:31.368203 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:31.370192 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:31.370193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:31.370193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:31.370199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:31.371286 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.371623 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.371623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.371624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.371670 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:31.371817 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:31.371818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:31.371818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:31.371895 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:31.371975 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:31.372046 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 10:21:31.372099 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:21:31.372728 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 19832
I20260619 10:21:31.376973 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 19966
I20260619 10:21:31.381023 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20100
I20260619 10:21:31.384905 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 19738
2026-06-19T10:21:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1851 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[20243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[20243](info): set up 2 sockets
Jun 19 10:21:31 dist-test-slave-xd2q krb5kdc[20243](info): commencing operation
krb5kdc: starting...
W20260619 10:21:32.406747 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:32.516176 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35517
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:32.592334 20256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:32.592535 20256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:32.592568 20256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:32.593652 20256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:32.593710 20256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:32.593734 20256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:32.593762 20256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:32.593789 20256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:32.595237 20256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:32.595496 20256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:32.595707 20256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:32.600104 20264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:32.600104 20262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:32.600135 20261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:32.600313 20256 server_base.cc:1034] running on GCE node
I20260619 10:21:32.600562 20256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:32.600821 20256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:32.601974 20256 hybrid_clock.cc:648] HybridClock initialized: now 1781864492601960 us; error 24 us; skew 500 ppm
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:32.604599 20256 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:32.604939 20256 webserver.cc:460] Webserver started at http://127.19.65.190:44359/ using document root <none> and password file <none>
I20260619 10:21:32.605123 20256 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:32.605162 20256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:32.605242 20256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:32.606165 20256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "3235044446454dbdade89251baea28dc"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "0398b2a2aaa69c0b0742db7a117ccf3b"
server_key_iv: "35121b3004c33bbd578f48044749ae49"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.606455 20256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "3235044446454dbdade89251baea28dc"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "0398b2a2aaa69c0b0742db7a117ccf3b"
server_key_iv: "35121b3004c33bbd578f48044749ae49"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.607645 20256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 10:21:32.608273 20271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.608450 20256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.608537 20256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "3235044446454dbdade89251baea28dc"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "0398b2a2aaa69c0b0742db7a117ccf3b"
server_key_iv: "35121b3004c33bbd578f48044749ae49"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.608620 20256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:32.625582 20256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:32.627170 20256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:32.627295 20256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:32.630745 20256 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:35517
I20260619 10:21:32.630810 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:35517 every 8 connection(s)
I20260619 10:21:32.631121 20256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:32.631937 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:32.634183 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc: Bootstrap starting.
I20260619 10:21:32.634737 20324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:32.635017 20324 log.cc:824] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:32.635716 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc: No bootstrap required, opened a new log
I20260619 10:21:32.637238 20324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } }
I20260619 10:21:32.637358 20324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:32.637379 20324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3235044446454dbdade89251baea28dc, State: Initialized, Role: FOLLOWER
I20260619 10:21:32.637553 20324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } }
I20260619 10:21:32.637621 20324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:32.637641 20324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:32.637705 20324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:32.638177 20324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } }
I20260619 10:21:32.638283 20324 leader_election.cc:302] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3235044446454dbdade89251baea28dc; no voters:
I20260619 10:21:32.638460 20324 leader_election.cc:288] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:32.638499 20327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:32.638646 20327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [term 1 LEADER]: Becoming Leader. State: Replica: 3235044446454dbdade89251baea28dc, State: Running, Role: LEADER
I20260619 10:21:32.638698 20324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:32.638728 20327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } }
I20260619 10:21:32.639104 20328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3235044446454dbdade89251baea28dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } } }
I20260619 10:21:32.639195 20328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:32.639361 20329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3235044446454dbdade89251baea28dc. Latest consensus state: current_term: 1 leader_uuid: "3235044446454dbdade89251baea28dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3235044446454dbdade89251baea28dc" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35517 } } }
I20260619 10:21:32.639478 20329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:32.639662 20335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:32.640149 20335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:32.640358 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20256
I20260619 10:21:32.640436 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:32.640583 19718 external_mini_cluster.cc:1428] Setting key 29b29888808cb6212d68f1503b56e511
I20260619 10:21:32.642203 20335 catalog_manager.cc:1347] Generated new cluster ID: ce65176fbe6c47719edc6cb7c0fa3a27
I20260619 10:21:32.642264 20335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:32.651212 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:32.641177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.19.65.190:35517)
0619 10:21:32.641613 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:32.641622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:32.641651 (+ 29us) server_negotiation.cc:407] Connection header received
0619 10:21:32.641779 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:32.641786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:32.641836 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:32.641959 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:32.642651 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.643594 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.644320 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.645059 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.645230 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:32.647784 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:32.647789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:32.647789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:32.647801 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:32.649563 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:32.649959 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:32.649963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:32.649963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:32.650064 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:32.650292 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:32.650293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:32.650293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:32.650527 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:32.650708 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:32.650781 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:21:32.651039 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":392,"thread_start_us":80,"threads_started":1}
I20260619 10:21:32.677675 20335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:32.678241 20335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:32.682216 20335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3235044446454dbdade89251baea28dc: Generated new TSK 0
I20260619 10:21:32.682396 20335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:32.714177 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:32.819780 20350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:32.819994 20350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:32.820015 20350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:32.821038 20350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:32.821069 20350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:32.821094 20350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:32.822386 20350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:32.822541 20350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:32.822727 20350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:21:32.827554 20350 server_base.cc:1034] running on GCE node
W20260619 10:21:32.827446 20355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:32.827443 20356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:32.827446 20358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:32.827944 20350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:32.828146 20350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:32.829315 20350 hybrid_clock.cc:648] HybridClock initialized: now 1781864492829301 us; error 34 us; skew 500 ppm
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:32.831897 20350 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:32.832451 20350 webserver.cc:460] Webserver started at http://127.19.65.129:45791/ using document root <none> and password file <none>
I20260619 10:21:32.832692 20350 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:32.832770 20350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:32.832891 20350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:32.834362 20350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "c0c11eeceed54100a613d40c00a295d4"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "bdf6b8e2cff0521d408538bf1ea6db26"
server_key_iv: "c9565b25f593f19c79f9cfaf282e00a9"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.834760 20350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "c0c11eeceed54100a613d40c00a295d4"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "bdf6b8e2cff0521d408538bf1ea6db26"
server_key_iv: "c9565b25f593f19c79f9cfaf282e00a9"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.836340 20350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 10:21:32.837288 20365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.837472 20350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:21:32.837563 20350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "c0c11eeceed54100a613d40c00a295d4"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "bdf6b8e2cff0521d408538bf1ea6db26"
server_key_iv: "c9565b25f593f19c79f9cfaf282e00a9"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.837631 20350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:32.865289 20350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:32.867698 20350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:32.867861 20350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:32.868106 20350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:32.868422 20350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:32.868481 20350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.868516 20350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:32.868542 20350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.874192 20350 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:43491
I20260619 10:21:32.874258 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:43491 every 8 connection(s)
I20260619 10:21:32.874648 20350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:32.879396 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20350
I20260619 10:21:32.879496 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:32.879654 19718 external_mini_cluster.cc:1428] Setting key 97dc92c8e5da78376aaf1295348cf10c
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:32.884562 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:32.875492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53333 (local address 127.19.65.190:35517)
0619 10:21:32.875534 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:32.875538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:32.875786 (+ 248us) server_negotiation.cc:407] Connection header received
0619 10:21:32.876740 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:32.876742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:32.876759 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:32.876822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:32.877450 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.877987 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.879000 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.879209 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.879313 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:32.881554 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:32.881555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:32.881555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:32.881566 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:32.883113 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:32.883607 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:32.883608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:32.883609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:32.883665 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:32.883935 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:32.883936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:32.883937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:32.884053 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:32.884129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:32.884388 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 10:21:32.884472 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:32.884764 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:32.875492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35517 (local address 127.19.65.129:53333)
0619 10:21:32.875666 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:32.875701 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:32.876628 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:32.876893 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:32.876898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:32.877158 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:32.877340 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:32.877344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.878083 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.878083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:32.878885 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:32.878887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:32.879322 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:32.879323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:32.879449 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:32.880017 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:32.880026 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:32.881458 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:32.883212 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:32.883218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:32.883220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:32.883515 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:32.883739 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:32.883740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:32.883741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:32.883861 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:32.884119 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:32.884125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:32.884261 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:21:32.884367 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 10:21:32.884526 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260619 10:21:32.885224 20479 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35517
I20260619 10:21:32.885324 20479 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:32.885493 20479 heartbeater.cc:511] Master 127.19.65.190:35517 requested a full tablet report, sending...
I20260619 10:21:32.886062 20288 ts_manager.cc:194] Registered new tserver with Master: c0c11eeceed54100a613d40c00a295d4 (127.19.65.129:43491)
I20260619 10:21:32.886981 20288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:53333
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:32.903863 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:32.980124 20484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:32.980305 20484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:32.980327 20484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:32.981510 20484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:32.981552 20484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:32.981604 20484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:32.983188 20484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:32.983461 20484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:32.983688 20484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:32.988462 20489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:32.988497 20484 server_base.cc:1034] running on GCE node
W20260619 10:21:32.988479 20492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:32.988479 20490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:32.988840 20484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:32.989032 20484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:32.990221 20484 hybrid_clock.cc:648] HybridClock initialized: now 1781864492990200 us; error 36 us; skew 500 ppm
Jun 19 10:21:32 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:32.992923 20484 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:32.993397 20484 webserver.cc:460] Webserver started at http://127.19.65.130:39567/ using document root <none> and password file <none>
I20260619 10:21:32.993634 20484 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:32.993682 20484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:32.993847 20484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:32.994864 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "6c4339774c244e47aeff5e7985e100de"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "4a4093e4ede7a8859379349f8d4936b3"
server_key_iv: "24801b28f39339be3b7c3bb4cb025451"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.995245 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "6c4339774c244e47aeff5e7985e100de"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "4a4093e4ede7a8859379349f8d4936b3"
server_key_iv: "24801b28f39339be3b7c3bb4cb025451"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.996585 20484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:32.997308 20499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:32.997475 20484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:32.997560 20484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "6c4339774c244e47aeff5e7985e100de"
format_stamp: "Formatted at 2026-06-19 10:21:32 on dist-test-slave-xd2q"
server_key: "4a4093e4ede7a8859379349f8d4936b3"
server_key_iv: "24801b28f39339be3b7c3bb4cb025451"
server_key_version: "encryptionkey@0"
I20260619 10:21:32.997649 20484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:33.006312 20484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:33.008081 20484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:33.008227 20484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:33.008412 20484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:33.008694 20484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:33.008724 20484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:33.008744 20484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:33.008790 20484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:33.014406 20484 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45821
I20260619 10:21:33.014503 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:45821 every 8 connection(s)
I20260619 10:21:33.014780 20484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:33.018550 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20484
I20260619 10:21:33.018638 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:33.018778 19718 external_mini_cluster.cc:1428] Setting key 606ab9cec7cd82afb9531eb5a7631c99
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:33.024170 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.015469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33263 (local address 127.19.65.190:35517)
0619 10:21:33.015532 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.015534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.015761 (+ 227us) server_negotiation.cc:407] Connection header received
0619 10:21:33.016597 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.016598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.016607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.016655 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.017155 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.017609 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.018505 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.018763 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.018842 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.021225 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.021226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.021227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.021233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.022750 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.023143 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.023144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.023145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.023224 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.023509 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.023510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.023510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.023649 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.023720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.023944 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.024040 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:21:33.024318 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.015487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35517 (local address 127.19.65.130:33263)
0619 10:21:33.015624 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.015656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.016518 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.016701 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.016704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.016920 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:33.017092 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.017095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.017679 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.017680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.018408 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.018410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.018871 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.018872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.019013 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.019568 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:33.019578 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:33.021128 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:33.022832 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:33.022837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:33.022838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:33.023069 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:33.023330 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:33.023331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:33.023331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:33.023435 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:33.023713 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:33.023717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:33.023814 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.023887 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.024087 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260619 10:21:33.024740 20613 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35517
I20260619 10:21:33.024819 20613 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:33.024997 20613 heartbeater.cc:511] Master 127.19.65.190:35517 requested a full tablet report, sending...
I20260619 10:21:33.025362 20288 ts_manager.cc:194] Registered new tserver with Master: 6c4339774c244e47aeff5e7985e100de (127.19.65.130:45821)
I20260619 10:21:33.025867 20288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:33263
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:33.041949 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:33.117728 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:33.117910 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:33.117933 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:33.118950 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:33.118983 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:33.119014 20618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:33.120560 20618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35517
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:33.120728 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:33.120882 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:33.125555 20623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:33.125559 20624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:33.125643 20626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:33.125692 20618 server_base.cc:1034] running on GCE node
I20260619 10:21:33.125957 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:33.126140 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:33.127270 20618 hybrid_clock.cc:648] HybridClock initialized: now 1781864493127253 us; error 33 us; skew 500 ppm
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:33.129717 20618 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:33.130137 20618 webserver.cc:460] Webserver started at http://127.19.65.131:46319/ using document root <none> and password file <none>
I20260619 10:21:33.130363 20618 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:33.130409 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:33.130553 20618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:33.131606 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "e6cee1554f3e40c0879926120db46f6e"
format_stamp: "Formatted at 2026-06-19 10:21:33 on dist-test-slave-xd2q"
server_key: "e47d6922ef3d1a7b7e64d58acb8c94b6"
server_key_iv: "50a5f14eeba15e7c45be9e03bc4086c0"
server_key_version: "encryptionkey@0"
I20260619 10:21:33.132046 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "e6cee1554f3e40c0879926120db46f6e"
format_stamp: "Formatted at 2026-06-19 10:21:33 on dist-test-slave-xd2q"
server_key: "e47d6922ef3d1a7b7e64d58acb8c94b6"
server_key_iv: "50a5f14eeba15e7c45be9e03bc4086c0"
server_key_version: "encryptionkey@0"
I20260619 10:21:33.133293 20618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:33.133994 20633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:33.134157 20618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:33.134248 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "e6cee1554f3e40c0879926120db46f6e"
format_stamp: "Formatted at 2026-06-19 10:21:33 on dist-test-slave-xd2q"
server_key: "e47d6922ef3d1a7b7e64d58acb8c94b6"
server_key_iv: "50a5f14eeba15e7c45be9e03bc4086c0"
server_key_version: "encryptionkey@0"
I20260619 10:21:33.134323 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:33.160212 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:33.161911 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:33.162087 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:33.162323 20618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:33.162639 20618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:33.162693 20618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:33.162731 20618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:33.162757 20618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:33.168428 20618 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:34567
I20260619 10:21:33.168474 20746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:34567 every 8 connection(s)
I20260619 10:21:33.168763 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:33.176164 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20618
I20260619 10:21:33.176260 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:33.176445 19718 external_mini_cluster.cc:1428] Setting key ce574308c5173051544effa0e1a6be9c
I20260619 10:21:33.177315 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.169404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59067 (local address 127.19.65.190:35517)
0619 10:21:33.169442 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.169444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.169655 (+ 211us) server_negotiation.cc:407] Connection header received
0619 10:21:33.170541 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.170542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.170552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.170600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.171136 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.171625 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.172489 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.172652 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.172696 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.174591 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.174592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.174592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.174598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.175835 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.176351 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.176352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.176352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.176400 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.176690 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.176691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.176691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.176805 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.176880 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.177143 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.177231 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:33.177502 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.169394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35517 (local address 127.19.65.131:59067)
0619 10:21:33.169561 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.169593 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.170444 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.170654 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.170657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.170890 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:33.171070 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.171074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.171708 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.171709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.172421 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.172423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.172716 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.172717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.172804 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.173205 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:33.173212 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:33.174485 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:33.175904 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:33.175909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:33.175911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:33.176242 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:33.176465 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:33.176466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:33.176466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:33.176603 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:33.176885 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:33.176891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:33.177015 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.177121 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.177279 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260619 10:21:33.178020 20747 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35517
I20260619 10:21:33.178133 20747 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:33.178299 20747 heartbeater.cc:511] Master 127.19.65.190:35517 requested a full tablet report, sending...
I20260619 10:21:33.178669 20288 ts_manager.cc:194] Registered new tserver with Master: e6cee1554f3e40c0879926120db46f6e (127.19.65.131:34567)
I20260619 10:21:33.179040 20288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:59067
I20260619 10:21:33.188016 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:21:33.194851 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.189450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.19.65.190:35517)
0619 10:21:33.189523 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.189524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.189588 (+ 64us) server_negotiation.cc:407] Connection header received
0619 10:21:33.189626 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.189627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.189635 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.189680 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.190190 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.190640 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.191280 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.191497 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.191548 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.192628 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.192629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.192629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.192634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.193984 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.194261 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.194262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.194262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.194308 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.194449 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.194449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.194449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.194565 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.194634 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.194716 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.194767 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:21:33.196808 20288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:33.198390 20288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:33.204907 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.200656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:45821 (local address 127.0.0.1:38870)
0619 10:21:33.201304 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.201312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.201381 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.201621 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.201622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.201624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:33.201665 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.201666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202854 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.202855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.203834 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.203836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.204575 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.204662 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.204681 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.204768 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.204817 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260619 10:21:33.205034 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.200640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:34567 (local address 127.0.0.1:35662)
0619 10:21:33.200794 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.200829 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.200890 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.201327 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.201330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.201333 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:33.201398 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.201399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202668 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.202669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.203836 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.203838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.204766 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.204819 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.204822 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.204880 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.204911 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
I20260619 10:21:33.205204 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.201009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.19.65.131:34567)
0619 10:21:33.201169 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.201173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.201181 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:33.201215 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.201217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.201236 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.201323 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:33.201478 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202568 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.203926 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.204680 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204757 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.204809 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.204900 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.204932 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260619 10:21:33.205230 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.200874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.19.65.130:45821)
0619 10:21:33.201024 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.201028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.201356 (+ 328us) server_negotiation.cc:407] Connection header received
0619 10:21:33.201456 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.201458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.201473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.201545 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:33.201742 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202764 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.203913 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.204464 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.204626 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.204954 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.205166 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260619 10:21:33.205574 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.200639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:43491 (local address 127.0.0.1:47742)
0619 10:21:33.201158 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.201169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.201224 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.201475 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.201476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.201478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:33.201529 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.201530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202492 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.202494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.203958 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.203959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.205349 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.205350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.205409 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.205412 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.205462 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.205508 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":63,"thread_start_us":78,"threads_started":1}
I20260619 10:21:33.205955 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.200818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.19.65.129:43491)
0619 10:21:33.200981 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.200985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.201220 (+ 235us) server_negotiation.cc:407] Connection header received
0619 10:21:33.201292 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.201295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.201311 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.201385 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:33.201605 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.202355 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204037 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.204682 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.204757 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.204809 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.205823 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.205865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260619 10:21:33.205859 20547 tablet_service.cc:1467] Processing CreateTablet for tablet 0853d30eee4043599e8d116757649d28 (DEFAULT_TABLE table=test-table [id=18c01577fabb48d8bc8b8577a542a7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:33.205858 20681 tablet_service.cc:1467] Processing CreateTablet for tablet 0853d30eee4043599e8d116757649d28 (DEFAULT_TABLE table=test-table [id=18c01577fabb48d8bc8b8577a542a7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:33.206182 20547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0853d30eee4043599e8d116757649d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:33.206182 20681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0853d30eee4043599e8d116757649d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:33.207842 20413 tablet_service.cc:1467] Processing CreateTablet for tablet 0853d30eee4043599e8d116757649d28 (DEFAULT_TABLE table=test-table [id=18c01577fabb48d8bc8b8577a542a7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:33.208179 20413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0853d30eee4043599e8d116757649d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:33.209436 20765 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Bootstrap starting.
I20260619 10:21:33.209685 20764 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Bootstrap starting.
I20260619 10:21:33.210331 20764 tablet_bootstrap.cc:654] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:33.210958 20764 log.cc:824] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:33.211486 20767 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Bootstrap starting.
I20260619 10:21:33.211793 20765 tablet_bootstrap.cc:654] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:33.211855 20764 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: No bootstrap required, opened a new log
I20260619 10:21:33.211921 20764 ts_tablet_manager.cc:1397] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:33.212150 20767 tablet_bootstrap.cc:654] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:33.212337 20765 log.cc:824] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:33.212670 20767 log.cc:824] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:33.213083 20765 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: No bootstrap required, opened a new log
I20260619 10:21:33.213148 20765 ts_tablet_manager.cc:1397] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:21:33.213379 20767 tablet_bootstrap.cc:492] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: No bootstrap required, opened a new log
I20260619 10:21:33.213447 20767 ts_tablet_manager.cc:1397] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:33.213944 20764 raft_consensus.cc:348] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.214116 20764 raft_consensus.cc:374] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:33.214151 20764 raft_consensus.cc:729] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4339774c244e47aeff5e7985e100de, State: Initialized, Role: FOLLOWER
I20260619 10:21:33.214301 20764 consensus_queue.cc:260] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.214572 20764 ts_tablet_manager.cc:1428] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:33.214684 20613 heartbeater.cc:503] Master 127.19.65.190:35517 was elected leader, sending a full tablet report...
I20260619 10:21:33.214998 20765 raft_consensus.cc:348] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.215173 20765 raft_consensus.cc:374] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:33.215215 20765 raft_consensus.cc:729] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cee1554f3e40c0879926120db46f6e, State: Initialized, Role: FOLLOWER
I20260619 10:21:33.215346 20765 consensus_queue.cc:260] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.215457 20767 raft_consensus.cc:348] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.215616 20747 heartbeater.cc:503] Master 127.19.65.190:35517 was elected leader, sending a full tablet report...
I20260619 10:21:33.215631 20767 raft_consensus.cc:374] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:33.215624 20765 ts_tablet_manager.cc:1428] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:21:33.215669 20767 raft_consensus.cc:729] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c11eeceed54100a613d40c00a295d4, State: Initialized, Role: FOLLOWER
I20260619 10:21:33.215793 20767 consensus_queue.cc:260] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.216051 20479 heartbeater.cc:503] Master 127.19.65.190:35517 was elected leader, sending a full tablet report...
I20260619 10:21:33.216051 20767 ts_tablet_manager.cc:1428] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 10:21:33.265329 20614 tablet.cc:2369] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:33.375716 20480 tablet.cc:2369] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:33.419283 20748 tablet.cc:2369] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:33.438081 20772 raft_consensus.cc:482] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:33.438202 20772 raft_consensus.cc:504] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.438755 20772 leader_election.cc:288] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6cee1554f3e40c0879926120db46f6e (127.19.65.131:34567), 6c4339774c244e47aeff5e7985e100de (127.19.65.130:45821)
I20260619 10:21:33.442446 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.438962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:47989 (local address 127.19.65.131:34567)
0619 10:21:33.439021 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.439023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.439472 (+ 449us) server_negotiation.cc:407] Connection header received
0619 10:21:33.439586 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.439587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.439616 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.439647 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:33.439847 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.440541 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.441760 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.442135 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.442171 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.442177 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.442341 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.442385 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:21:33.442458 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.438910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:34567 (local address 127.19.65.129:47989)
0619 10:21:33.439165 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.439171 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.439501 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.439713 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.439713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.439715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:33.439786 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.439788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.440633 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.440634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.441658 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.441660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.442220 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.442221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.442268 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.442271 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.442335 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.442377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":53,"threads_started":1}
I20260619 10:21:33.442895 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.438966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:32955 (local address 127.19.65.130:45821)
0619 10:21:33.439068 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.439070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.439087 (+ 17us) server_negotiation.cc:407] Connection header received
0619 10:21:33.439154 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.439155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.439165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.439240 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:33.439593 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.440346 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.441746 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.442504 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.442611 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.442622 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.442787 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.442829 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 10:21:33.443005 20701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0853d30eee4043599e8d116757649d28" candidate_uuid: "c0c11eeceed54100a613d40c00a295d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6cee1554f3e40c0879926120db46f6e" is_pre_election: true
I20260619 10:21:33.443149 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.438907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:45821 (local address 127.19.65.129:32955)
0619 10:21:33.439028 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:33.439039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:33.439093 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:33.439297 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:33.439298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:33.439301 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:33.439494 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.439497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.440459 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.440459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.441656 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:33.441658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.442609 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.442610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:33.442696 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.442701 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:21:33.443021 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 10:21:33.443062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260619 10:21:33.443162 20701 raft_consensus.cc:2454] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c11eeceed54100a613d40c00a295d4 in term 0.
I20260619 10:21:33.443463 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0853d30eee4043599e8d116757649d28" candidate_uuid: "c0c11eeceed54100a613d40c00a295d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4339774c244e47aeff5e7985e100de" is_pre_election: true
I20260619 10:21:33.443578 20567 raft_consensus.cc:2454] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c11eeceed54100a613d40c00a295d4 in term 0.
I20260619 10:21:33.443760 20367 leader_election.cc:302] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0c11eeceed54100a613d40c00a295d4, e6cee1554f3e40c0879926120db46f6e; no voters:
I20260619 10:21:33.443902 20772 raft_consensus.cc:2781] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:33.443989 20772 raft_consensus.cc:482] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:33.444016 20772 raft_consensus.cc:3037] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:33.444720 20772 raft_consensus.cc:504] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.444859 20772 leader_election.cc:288] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [CANDIDATE]: Term 1 election: Requested vote from peers e6cee1554f3e40c0879926120db46f6e (127.19.65.131:34567), 6c4339774c244e47aeff5e7985e100de (127.19.65.130:45821)
I20260619 10:21:33.444994 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0853d30eee4043599e8d116757649d28" candidate_uuid: "c0c11eeceed54100a613d40c00a295d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4339774c244e47aeff5e7985e100de"
I20260619 10:21:33.445075 20567 raft_consensus.cc:3037] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:33.445145 20701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0853d30eee4043599e8d116757649d28" candidate_uuid: "c0c11eeceed54100a613d40c00a295d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6cee1554f3e40c0879926120db46f6e"
I20260619 10:21:33.445247 20701 raft_consensus.cc:3037] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:33.445825 20567 raft_consensus.cc:2454] T 0853d30eee4043599e8d116757649d28 P 6c4339774c244e47aeff5e7985e100de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c11eeceed54100a613d40c00a295d4 in term 1.
I20260619 10:21:33.445988 20366 leader_election.cc:302] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4339774c244e47aeff5e7985e100de, c0c11eeceed54100a613d40c00a295d4; no voters:
I20260619 10:21:33.446081 20772 raft_consensus.cc:2781] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:33.446206 20772 raft_consensus.cc:686] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [term 1 LEADER]: Becoming Leader. State: Replica: c0c11eeceed54100a613d40c00a295d4, State: Running, Role: LEADER
I20260619 10:21:33.446273 20701 raft_consensus.cc:2454] T 0853d30eee4043599e8d116757649d28 P e6cee1554f3e40c0879926120db46f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c11eeceed54100a613d40c00a295d4 in term 1.
I20260619 10:21:33.446295 20772 consensus_queue.cc:237] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } }
I20260619 10:21:33.447079 20288 catalog_manager.cc:5495] T 0853d30eee4043599e8d116757649d28 P c0c11eeceed54100a613d40c00a295d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0c11eeceed54100a613d40c00a295d4 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "c0c11eeceed54100a613d40c00a295d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cee1554f3e40c0879926120db46f6e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c11eeceed54100a613d40c00a295d4" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 43491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c4339774c244e47aeff5e7985e100de" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45821 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:33.491866 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.484533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.19.65.129:43491)
0619 10:21:33.484605 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.484607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.484660 (+ 53us) server_negotiation.cc:407] Connection header received
0619 10:21:33.484739 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.484740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.484750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.484822 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.485375 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.485908 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.486584 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.486787 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.486863 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.489337 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.489347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.489347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.489353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.490909 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.491201 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.491203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.491204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.491261 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.491455 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.491456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.491456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.491583 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.491681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.491744 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.491803 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:33.513012 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.506299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.19.65.129:43491)
0619 10:21:33.506354 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.506356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.506453 (+ 97us) server_negotiation.cc:407] Connection header received
0619 10:21:33.506545 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.506546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.506565 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.506626 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.507149 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.507661 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.508350 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.508533 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.508603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.510552 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.510553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.510554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.510561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.511908 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.512231 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.512232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.512233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.512282 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.512469 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.512469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.512470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.512559 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.512673 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.512836 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.512906 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:33.532886 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:33.526961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.19.65.129:43491)
0619 10:21:33.527030 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:33.527031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:33.527103 (+ 72us) server_negotiation.cc:407] Connection header received
0619 10:21:33.527138 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:33.527138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:33.527146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:33.527191 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:33.527713 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.528149 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.528779 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:33.528943 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:33.528989 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:33.530904 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:33.530905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:33.530905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:33.530911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:33.532082 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.532353 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.532353 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.532354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.532396 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:33.532535 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:33.532535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:33.532536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:33.532606 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:33.532690 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:33.532749 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 10:21:33.532799 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 10:21:33.533356 20393 tablet_service.cc:2908] Rejecting scan request for tablet 0853d30eee4043599e8d116757649d28: Uninitialized: clean time has not yet been initialized
I20260619 10:21:33.534107 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20350
I20260619 10:21:33.538993 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20484
I20260619 10:21:33.543717 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20618
I20260619 10:21:33.547652 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20256
2026-06-19T10:21:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2163 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20789](info): set up 2 sockets
Jun 19 10:21:33 dist-test-slave-xd2q krb5kdc[20789](info): commencing operation
krb5kdc: starting...
W20260619 10:21:34.558763 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:34 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:34.667874 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44885
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:34.744535 20802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:34.744755 20802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:34.744786 20802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:34.745878 20802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:34.745954 20802 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:21:34.745982 20802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:34.746013 20802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:34.746037 20802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:34.746062 20802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:34.747476 20802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:34.747726 20802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:34.747931 20802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:21:34.752596 20802 server_base.cc:1034] running on GCE node
W20260619 10:21:34.752526 20807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:34.752499 20808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:34.752501 20810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:34.753104 20802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:34.753371 20802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:34.754520 20802 hybrid_clock.cc:648] HybridClock initialized: now 1781864494754517 us; error 33 us; skew 500 ppm
Jun 19 10:21:34 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:34.763429 20802 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:34.763844 20802 webserver.cc:460] Webserver started at http://127.19.65.190:43457/ using document root <none> and password file <none>
I20260619 10:21:34.764019 20802 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:34.764056 20802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:34.764139 20802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:34.765187 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "1e56c5693a604d18b451072d71f82a28"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "3fc66e18083554f12ce5d2647496f61f"
server_key_iv: "63ea621979039b1ebfa9747f24892ad6"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.765471 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "1e56c5693a604d18b451072d71f82a28"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "3fc66e18083554f12ce5d2647496f61f"
server_key_iv: "63ea621979039b1ebfa9747f24892ad6"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.766655 20802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 10:21:34.767411 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:34.767549 20802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:34.767629 20802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "1e56c5693a604d18b451072d71f82a28"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "3fc66e18083554f12ce5d2647496f61f"
server_key_iv: "63ea621979039b1ebfa9747f24892ad6"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.767678 20802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:34.791270 20802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:34.793557 20802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:34.793691 20802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:34.798004 20802 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44885
I20260619 10:21:34.798094 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:44885 every 8 connection(s)
I20260619 10:21:34.798400 20802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:34.799054 20880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:34.801172 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Bootstrap starting.
I20260619 10:21:34.801680 20880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:34.801842 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20802
I20260619 10:21:34.801939 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:34.802024 20880 log.cc:824] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:34.802120 19718 external_mini_cluster.cc:1428] Setting key 15ec4432221f7edb06cff84e5ebcdc35
I20260619 10:21:34.802780 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: No bootstrap required, opened a new log
I20260619 10:21:34.805025 20880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } }
I20260619 10:21:34.805183 20880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:34.805214 20880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e56c5693a604d18b451072d71f82a28, State: Initialized, Role: FOLLOWER
I20260619 10:21:34.805330 20880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } }
I20260619 10:21:34.805395 20880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:34.805423 20880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:34.805469 20880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:34.806077 20880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } }
I20260619 10:21:34.806172 20880 leader_election.cc:302] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e56c5693a604d18b451072d71f82a28; no voters:
I20260619 10:21:34.806355 20880 leader_election.cc:288] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:34.806378 20885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:34.806524 20885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [term 1 LEADER]: Becoming Leader. State: Replica: 1e56c5693a604d18b451072d71f82a28, State: Running, Role: LEADER
I20260619 10:21:34.806597 20880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:34.806599 20885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } }
I20260619 10:21:34.806955 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e56c5693a604d18b451072d71f82a28. Latest consensus state: current_term: 1 leader_uuid: "1e56c5693a604d18b451072d71f82a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } } }
I20260619 10:21:34.807034 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:34.807222 20891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:34.807690 20891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:34.809345 20886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e56c5693a604d18b451072d71f82a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e56c5693a604d18b451072d71f82a28" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44885 } } }
I20260619 10:21:34.809430 20886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28 [sys.catalog]: This master's current role is: LEADER
Jun 19 10:21:34 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:34.812505 20891 catalog_manager.cc:1347] Generated new cluster ID: b62b28471c6e440cb963fa61886764a5
I20260619 10:21:34.812553 20891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:34.813448 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:34.802612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.19.65.190:44885)
0619 10:21:34.802782 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:34.802790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:34.802822 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:21:34.803323 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:34.803331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:34.803428 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:34.803575 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:34.804147 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:34.805963 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:34.806930 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:34.807565 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:34.807664 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:34.810267 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:34.810271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:34.810271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:34.810282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:34.812052 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:34.812386 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:34.812389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:34.812389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:34.812479 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:34.812672 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:34.812673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:34.812673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:34.812902 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:34.813060 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:34.813151 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 10:21:34.813284 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260619 10:21:34.847028 20891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:34.847739 20891 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:34.853217 20891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Generated new TSK 0
I20260619 10:21:34.853380 20891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:34.881479 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44885
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:34.959271 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:34.959508 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:34.959530 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:34.960655 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:34.960692 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:34.960711 20906 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:34.960765 20906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:34.962054 20906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:34.962278 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:34.962498 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:34.967170 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:34.967156 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:34.967150 20906 server_base.cc:1034] running on GCE node
W20260619 10:21:34.967193 20914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:34.967485 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:34.967767 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:34.968919 20906 hybrid_clock.cc:648] HybridClock initialized: now 1781864494968900 us; error 41 us; skew 500 ppm
Jun 19 10:21:34 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:34.971266 20906 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:34.971684 20906 webserver.cc:460] Webserver started at http://127.19.65.129:43967/ using document root <none> and password file <none>
I20260619 10:21:34.971902 20906 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:34.971962 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:34.972105 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:34.972991 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "89e28b203843431cb21542ed02474534"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "209fe52fd5d5168468d1f77c83cd86fc"
server_key_iv: "c0537f98d242bdff67eb5c630586c716"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.973318 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "89e28b203843431cb21542ed02474534"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "209fe52fd5d5168468d1f77c83cd86fc"
server_key_iv: "c0537f98d242bdff67eb5c630586c716"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.974527 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:34.975207 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:34.975394 20906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:34.975485 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "89e28b203843431cb21542ed02474534"
format_stamp: "Formatted at 2026-06-19 10:21:34 on dist-test-slave-xd2q"
server_key: "209fe52fd5d5168468d1f77c83cd86fc"
server_key_iv: "c0537f98d242bdff67eb5c630586c716"
server_key_version: "encryptionkey@0"
I20260619 10:21:34.975567 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:34.995376 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:34.997114 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:34.997262 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:34.997706 20906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:34.997766 20906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:34.997807 20906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:34.997831 20906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:35.004988 20906 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:37223
I20260619 10:21:35.005103 21036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:37223 every 8 connection(s)
I20260619 10:21:35.005343 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:35.005609 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 20906
I20260619 10:21:35.005712 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:35.005882 19718 external_mini_cluster.cc:1428] Setting key 0ab5cf05ffff3cae42fbdd56a9e7acd6
I20260619 10:21:35.008437 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:34.998214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:32891 (local address 127.19.65.190:44885)
0619 10:21:34.998279 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:34.998282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:34.998545 (+ 263us) server_negotiation.cc:407] Connection header received
0619 10:21:34.999240 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:34.999242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:34.999251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:34.999291 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.000303 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.000788 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.001825 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.002040 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.002119 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.004585 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.004586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.004587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.004595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.006841 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.007277 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.007278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.007278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.007352 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.007699 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.007700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.007700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.007835 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.007930 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.008223 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.008348 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:21:35.008596 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:34.998193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.129:32891)
0619 10:21:34.998380 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:34.998424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:34.999126 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:34.999413 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:34.999418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:34.999849 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.000199 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.000206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.000885 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.000886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.001718 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.001721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.002130 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.002131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.002250 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.002723 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.002731 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.004492 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.006940 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.006946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.006949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.007188 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.007424 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.007425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.007425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.007605 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.007923 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.007929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.008078 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.008187 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.008365 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260619 10:21:35.009069 21037 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44885
I20260619 10:21:35.009143 21037 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:35.009315 21037 heartbeater.cc:511] Master 127.19.65.190:44885 requested a full tablet report, sending...
I20260619 10:21:35.009820 20833 ts_manager.cc:194] Registered new tserver with Master: 89e28b203843431cb21542ed02474534 (127.19.65.129:37223)
I20260619 10:21:35.010779 20833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:32891
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:21:35.016038 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.009994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.129:53309)
0619 10:21:35.010123 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.010133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.010207 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.010323 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.010324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.010491 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.010535 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.010536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.011340 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.011342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.012315 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.012318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.012661 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.012662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.012738 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.013109 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.013114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.013498 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.015014 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.015015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.015015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.015196 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.015420 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.015421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.015421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.015467 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.015822 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.015822 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.015857 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.015907 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.015979 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260619 10:21:35.016039 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.010039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53309 (local address 127.19.65.190:44885)
0619 10:21:35.010074 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.010076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.010185 (+ 109us) server_negotiation.cc:407] Connection header received
0619 10:21:35.010256 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.010257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.010266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.010319 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.010615 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.011214 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.012408 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.012583 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.012640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.013588 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.013589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.013589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.013595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.014939 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.015259 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.015260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.015260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.015344 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.015584 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.015585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.015585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.015754 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.015816 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.015927 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.015985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:35.030018 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44885
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:35.105859 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:35.106117 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:35.106158 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:35.107477 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:35.107532 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:35.107568 21046 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:35.107601 21046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:35.109172 21046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:35.109406 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:35.109617 21046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:35.114321 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:35.114337 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:35.114338 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:35.114393 21046 server_base.cc:1034] running on GCE node
I20260619 10:21:35.114660 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:35.114848 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:35.115983 21046 hybrid_clock.cc:648] HybridClock initialized: now 1781864495115948 us; error 48 us; skew 500 ppm
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:35.118194 21046 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:35.118530 21046 webserver.cc:460] Webserver started at http://127.19.65.130:40155/ using document root <none> and password file <none>
I20260619 10:21:35.118698 21046 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:35.118734 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:35.118809 21046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:35.119707 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "35d9a64bb626fba90b8407b4b68dee48"
server_key_iv: "fd6f7d025c85ff3171cc95dcbb6451df"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.120005 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "35d9a64bb626fba90b8407b4b68dee48"
server_key_iv: "fd6f7d025c85ff3171cc95dcbb6451df"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.121146 21046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:35.121742 21061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.121878 21046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:35.121951 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "35d9a64bb626fba90b8407b4b68dee48"
server_key_iv: "fd6f7d025c85ff3171cc95dcbb6451df"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.122004 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:35.144874 21046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:35.146452 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:35.146545 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:35.146904 21046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:35.146932 21046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.146986 21046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:35.147010 21046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.153666 21046 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:36297
I20260619 10:21:35.153720 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:36297 every 8 connection(s)
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:35.153991 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:35.154276 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21046
I20260619 10:21:35.154368 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:35.154563 19718 external_mini_cluster.cc:1428] Setting key 1ff38c619c0cd18321ae2d9e9ca7c462
I20260619 10:21:35.157315 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.147358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:49853 (local address 127.19.65.190:44885)
0619 10:21:35.147448 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.147450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.147673 (+ 223us) server_negotiation.cc:407] Connection header received
0619 10:21:35.148309 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.148310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.148321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.148372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.149230 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.149953 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.151488 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.151791 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.151855 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.154548 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.154550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.154550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.154557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.155986 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.156411 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.156413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.156413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.156481 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.156754 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.156754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.156755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.156851 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.156901 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.157205 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.157261 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 10:21:35.157538 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.147325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.130:49853)
0619 10:21:35.147545 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.147588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.148201 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.148461 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.148466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.148846 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.149108 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.149114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.150046 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.150047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.151399 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.151402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.151883 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.151884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.152029 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.152628 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.152639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.154463 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.156069 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.156075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.156077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.156320 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.156559 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.156560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.156560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.156688 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.156911 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.156916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.157063 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.157186 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.157340 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260619 10:21:35.157977 21177 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44885
I20260619 10:21:35.158049 21177 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:35.158206 21177 heartbeater.cc:511] Master 127.19.65.190:44885 requested a full tablet report, sending...
I20260619 10:21:35.158550 20834 ts_manager.cc:194] Registered new tserver with Master: 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297)
I20260619 10:21:35.159036 20834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:49853
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:21:35.164302 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.158939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52211 (local address 127.19.65.190:44885)
0619 10:21:35.158971 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.158973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.159086 (+ 113us) server_negotiation.cc:407] Connection header received
0619 10:21:35.159176 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.159177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.159185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.159210 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.159590 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.159997 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.160661 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.160851 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.160917 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.161767 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.161769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.161769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.161775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.163073 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.163523 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.163525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.163525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.163570 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.163769 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.163770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.163770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.163906 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.163980 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.164146 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.164220 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:35.164316 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.158901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.130:52211)
0619 10:21:35.159019 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.159027 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.159098 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.159258 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.159258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.159452 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.159501 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.159502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.160078 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.160078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.160574 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.160576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.160922 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.160922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.160987 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.161381 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.161385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.161669 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.163166 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.163167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.163168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.163380 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.163636 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.163637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.163637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.163695 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.163985 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.163985 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.164040 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.164122 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.164216 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:35.177964 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44885
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:35.255206 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:35.255401 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:35.255424 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:35.256490 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:35.256520 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:35.256539 21186 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:35.256556 21186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:35.257881 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:35.258100 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:35.258262 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:35.262866 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:35.262939 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:35.262918 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:35.263114 21186 server_base.cc:1034] running on GCE node
I20260619 10:21:35.263343 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:35.263639 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:35.264830 21186 hybrid_clock.cc:648] HybridClock initialized: now 1781864495264812 us; error 25 us; skew 500 ppm
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:35.267060 21186 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:35.267578 21186 webserver.cc:460] Webserver started at http://127.19.65.131:40919/ using document root <none> and password file <none>
I20260619 10:21:35.267824 21186 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:35.267866 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:35.268014 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:35.269014 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "ed1863a798ad46e3acb6777583ce5dc2"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "0800f730ca50439dad771254f4841c77"
server_key_iv: "aa48288fe2a17cb3460b327f0141c501"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.269344 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "ed1863a798ad46e3acb6777583ce5dc2"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "0800f730ca50439dad771254f4841c77"
server_key_iv: "aa48288fe2a17cb3460b327f0141c501"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.270730 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:35.271485 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.271642 21186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:35.271742 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "ed1863a798ad46e3acb6777583ce5dc2"
format_stamp: "Formatted at 2026-06-19 10:21:35 on dist-test-slave-xd2q"
server_key: "0800f730ca50439dad771254f4841c77"
server_key_iv: "aa48288fe2a17cb3460b327f0141c501"
server_key_version: "encryptionkey@0"
I20260619 10:21:35.271839 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:35.292338 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:35.294579 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:35.294731 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:35.295241 21186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:35.295323 21186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.295365 21186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:35.295392 21186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:35.302241 21186 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45095
I20260619 10:21:35.302276 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45095 every 8 connection(s)
I20260619 10:21:35.302644 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:35.305235 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.295697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57323 (local address 127.19.65.190:44885)
0619 10:21:35.295772 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.295775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.295965 (+ 190us) server_negotiation.cc:407] Connection header received
0619 10:21:35.296426 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.296427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.296441 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.296520 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.297251 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.298082 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.299584 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.299920 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.300005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.302681 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.302683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.302683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.302690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.304128 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.304507 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.304508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.304509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.304555 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.304751 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.304751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.304751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.304854 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.304902 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.305119 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.305176 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:21:35.305374 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.295690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.131:57323)
0619 10:21:35.295836 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.295874 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.296326 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.296567 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.296570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.296879 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.297147 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.297153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.298189 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.298190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.299497 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.299499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.300029 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.300031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.300205 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.300694 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.300706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.302583 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.304196 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.304214 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.304215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.304437 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.304607 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.304607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.304607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.304697 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.304907 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.304910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.305012 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.305090 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.305196 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260619 10:21:35.305809 21317 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44885
I20260619 10:21:35.305886 21317 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:35.306082 21317 heartbeater.cc:511] Master 127.19.65.190:44885 requested a full tablet report, sending...
I20260619 10:21:35.306438 20833 ts_manager.cc:194] Registered new tserver with Master: ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095)
I20260619 10:21:35.306900 20833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:57323
I20260619 10:21:35.311995 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.306846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:33147 (local address 127.19.65.190:44885)
0619 10:21:35.306889 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.306891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.306962 (+ 71us) server_negotiation.cc:407] Connection header received
0619 10:21:35.307036 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.307036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.307045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.307125 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.307556 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.308131 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.308920 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.309127 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.309199 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.309941 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.309942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.309943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.309949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.311147 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.311499 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.311499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.311500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.311546 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.311693 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.311694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.311694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.311769 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.311809 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.311893 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.311941 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:21:35.312003 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.306784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.19.65.131:33147)
0619 10:21:35.306916 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.306926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.306974 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.307132 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.307133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.307427 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.307479 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.307480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.308217 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.308218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.308822 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.308824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.309215 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.309216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.309286 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.309617 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.309620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.309860 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.311230 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.311231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.311231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.311432 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.311607 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.311608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.311608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.311655 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.311801 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.311801 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.311829 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.311878 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.311943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260619 10:21:35.312439 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21186
I20260619 10:21:35.312558 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:35.312723 19718 external_mini_cluster.cc:1428] Setting key 222add1ae07a69b7875d387edeae365d
I20260619 10:21:35.313536 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:35.333881 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.327818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.19.65.190:44885)
0619 10:21:35.327871 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.327873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.327881 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:35.327950 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.327951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.327962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.328035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.328572 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.329029 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.329705 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.329881 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.329973 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.332006 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.332008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.332008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.332013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.333076 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.333355 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.333356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.333356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.333399 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.333536 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.333537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.333537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.333613 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.333676 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.333755 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.333803 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:21:35.335235 20833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:35.336763 20833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:35.342597 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.338783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:37223 (local address 127.0.0.1:35334)
0619 10:21:35.339056 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.339115 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.339183 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.339552 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.339553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.339554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.339594 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.339595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340435 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.340436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.341602 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.341604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342305 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.342384 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342396 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.342455 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.342499 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":56,"threads_started":1}
I20260619 10:21:35.342715 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.338887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.19.65.131:45095)
0619 10:21:35.339070 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.339075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.339263 (+ 188us) server_negotiation.cc:407] Connection header received
0619 10:21:35.339302 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.339352 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.339368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.339477 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.339764 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340757 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.341709 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342369 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342456 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342508 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.342584 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.342616 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260619 10:21:35.342803 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.338946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36297 (local address 127.0.0.1:36418)
0619 10:21:35.339127 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.339142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.339184 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.339389 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.339391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.339393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.339474 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.339476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340721 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.340722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.341730 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.341731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342584 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.342640 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342644 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.342693 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.342736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":63,"threads_started":1}
I20260619 10:21:35.342937 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.338882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.19.65.129:37223)
0619 10:21:35.339337 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.339340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.339349 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:35.339386 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.339388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.339402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.339489 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.339672 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340350 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.341692 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342218 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342296 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342351 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.342771 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.342812 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":68,"threads_started":1}
I20260619 10:21:35.343067 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.339045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.19.65.130:36297)
0619 10:21:35.339223 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.339227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.339237 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:35.339276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.339278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.339294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.339390 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.339531 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340640 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.341792 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342445 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342497 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342536 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.342925 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.342965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260619 10:21:35.343145 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.338767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45095 (local address 127.0.0.1:59282)
0619 10:21:35.339055 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.339115 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.339184 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.339645 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.339646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.339648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.339687 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.339688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.340836 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.340837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.341650 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.341652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.342456 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.342457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.342509 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.342511 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.342558 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.342605 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":45,"threads_started":1}
I20260619 10:21:35.343549 20971 tablet_service.cc:1467] Processing CreateTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 (DEFAULT_TABLE table=test-table [id=fa20807f9e644e1e87a1cacc876ba598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.343662 21111 tablet_service.cc:1467] Processing CreateTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 (DEFAULT_TABLE table=test-table [id=fa20807f9e644e1e87a1cacc876ba598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.343711 21251 tablet_service.cc:1467] Processing CreateTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 (DEFAULT_TABLE table=test-table [id=fa20807f9e644e1e87a1cacc876ba598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.343804 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd49f391d00a4f6683ca2972e8ee5371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.343964 21251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd49f391d00a4f6683ca2972e8ee5371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.343966 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd49f391d00a4f6683ca2972e8ee5371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.346307 21337 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Bootstrap starting.
I20260619 10:21:35.346864 21337 tablet_bootstrap.cc:654] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.347029 21338 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Bootstrap starting.
I20260619 10:21:35.347384 21337 log.cc:824] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:35.347748 21339 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Bootstrap starting.
I20260619 10:21:35.348001 21338 tablet_bootstrap.cc:654] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.348224 21337 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: No bootstrap required, opened a new log
I20260619 10:21:35.348302 21337 ts_tablet_manager.cc:1397] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:21:35.348352 21338 log.cc:824] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:35.348408 21339 tablet_bootstrap.cc:654] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.348701 21339 log.cc:824] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:35.349045 21338 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: No bootstrap required, opened a new log
I20260619 10:21:35.349114 21338 ts_tablet_manager.cc:1397] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:35.349371 21339 tablet_bootstrap.cc:492] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: No bootstrap required, opened a new log
I20260619 10:21:35.349445 21339 ts_tablet_manager.cc:1397] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:35.350276 21337 raft_consensus.cc:348] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.350435 21337 raft_consensus.cc:374] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.350471 21337 raft_consensus.cc:729] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e28b203843431cb21542ed02474534, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.350579 21337 consensus_queue.cc:260] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.350821 21337 ts_tablet_manager.cc:1428] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 10:21:35.350853 21037 heartbeater.cc:503] Master 127.19.65.190:44885 was elected leader, sending a full tablet report...
I20260619 10:21:35.351114 21338 raft_consensus.cc:348] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.351265 21338 raft_consensus.cc:374] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.351321 21338 raft_consensus.cc:729] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1863a798ad46e3acb6777583ce5dc2, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.351183 21339 raft_consensus.cc:348] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.351402 21339 raft_consensus.cc:374] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.351449 21339 raft_consensus.cc:729] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae6ba1914b2435bb76c46a7b85406b7, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.351433 21338 consensus_queue.cc:260] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.351593 21339 consensus_queue.cc:260] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.351694 21338 ts_tablet_manager.cc:1428] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:35.351814 21317 heartbeater.cc:503] Master 127.19.65.190:44885 was elected leader, sending a full tablet report...
I20260619 10:21:35.351915 21177 heartbeater.cc:503] Master 127.19.65.190:44885 was elected leader, sending a full tablet report...
I20260619 10:21:35.352192 21339 ts_tablet_manager.cc:1428] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:35.368902 21343 raft_consensus.cc:482] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:35.369004 21343 raft_consensus.cc:504] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.369335 21343 leader_election.cc:288] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095), 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297)
I20260619 10:21:35.372182 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.369487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45095 (local address 127.19.65.129:49975)
0619 10:21:35.369536 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.369546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.369585 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.369724 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.369725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.369726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.369788 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.369789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.370455 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.370456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.371251 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.371253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.371919 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.371920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.372011 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.372015 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.372074 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.372113 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:21:35.372186 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.369527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:49975 (local address 127.19.65.131:45095)
0619 10:21:35.369613 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.369615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.369624 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:35.369647 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.369647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.369655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.369712 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.369856 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.370381 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.371386 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.371850 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.371901 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.371908 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.372082 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.372107 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 10:21:35.372443 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.369551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33069 (local address 127.19.65.130:36297)
0619 10:21:35.369605 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.369607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.369758 (+ 151us) server_negotiation.cc:407] Connection header received
0619 10:21:35.369806 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.369807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.369817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.369881 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.370053 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.370567 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.371658 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.372082 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.372139 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.372146 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.372351 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.372387 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:21:35.372453 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.369504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36297 (local address 127.19.65.129:33069)
0619 10:21:35.369678 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.369689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.369746 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.369886 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.369887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.369901 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.369979 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.369980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.370660 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.370661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.371581 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.371584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.372173 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.372174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.372246 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.372250 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.372319 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.372378 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260619 10:21:35.372586 21271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd49f391d00a4f6683ca2972e8ee5371" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1863a798ad46e3acb6777583ce5dc2" is_pre_election: true
I20260619 10:21:35.372699 21271 raft_consensus.cc:2454] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 0.
I20260619 10:21:35.372761 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd49f391d00a4f6683ca2972e8ee5371" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" is_pre_election: true
I20260619 10:21:35.372896 21131 raft_consensus.cc:2454] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 0.
I20260619 10:21:35.372907 20923 leader_election.cc:302] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e28b203843431cb21542ed02474534, ed1863a798ad46e3acb6777583ce5dc2; no voters:
I20260619 10:21:35.373005 21343 raft_consensus.cc:2781] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:35.373085 21343 raft_consensus.cc:482] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:35.373118 21343 raft_consensus.cc:3037] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.373934 21343 raft_consensus.cc:504] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.374055 21343 leader_election.cc:288] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 election: Requested vote from peers ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095), 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297)
I20260619 10:21:35.374193 21271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd49f391d00a4f6683ca2972e8ee5371" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1863a798ad46e3acb6777583ce5dc2"
I20260619 10:21:35.374210 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd49f391d00a4f6683ca2972e8ee5371" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
I20260619 10:21:35.374266 21271 raft_consensus.cc:3037] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.374297 21131 raft_consensus.cc:3037] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.374943 21131 raft_consensus.cc:2454] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 1.
I20260619 10:21:35.375005 21271 raft_consensus.cc:2454] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 1.
I20260619 10:21:35.375180 20924 leader_election.cc:302] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae6ba1914b2435bb76c46a7b85406b7, 89e28b203843431cb21542ed02474534; no voters:
I20260619 10:21:35.375314 21343 raft_consensus.cc:2781] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:35.375451 21343 raft_consensus.cc:686] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 1 LEADER]: Becoming Leader. State: Replica: 89e28b203843431cb21542ed02474534, State: Running, Role: LEADER
I20260619 10:21:35.375555 21343 consensus_queue.cc:237] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } }
I20260619 10:21:35.376250 20834 catalog_manager.cc:5495] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e28b203843431cb21542ed02474534 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "89e28b203843431cb21542ed02474534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } health_report { overall_health: UNKNOWN } } }
I20260619 10:21:35.381752 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.379403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.19.65.129:37223)
0619 10:21:35.379454 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.379456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.379463 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:21:35.379537 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.379538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.379552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.379608 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:35.379735 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.380165 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.380909 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.381117 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.381197 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.381406 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.381507 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.381592 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.381645 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.381662 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:21:35.383170 21271 raft_consensus.cc:1261] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Refusing update from remote peer 89e28b203843431cb21542ed02474534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.383208 21131 raft_consensus.cc:1261] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Refusing update from remote peer 89e28b203843431cb21542ed02474534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.383472 21347 consensus_queue.cc:1035] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.383618 21343 consensus_queue.cc:1035] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.385756 21352 mvcc.cc:205] Tried to move back new op lower bound from 7298516973087469568 to 7298516973058838528. Current Snapshot: MvccSnapshot[applied={T|T < 7298516973087469568}]
I20260619 10:21:35.386300 21351 mvcc.cc:205] Tried to move back new op lower bound from 7298516973087469568 to 7298516973058838528. Current Snapshot: MvccSnapshot[applied={T|T < 7298516973087469568}]
I20260619 10:21:35.388918 20834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:35.389003 20834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:35.389650 20834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Sending DeleteTablet for 3 replicas of tablet cd49f391d00a4f6683ca2972e8ee5371
I20260619 10:21:35.389919 21251 tablet_service.cc:1514] Processing DeleteTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59282
I20260619 10:21:35.390089 20971 tablet_service.cc:1514] Processing DeleteTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35334
I20260619 10:21:35.390125 20834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:35.390259 20834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:35.390260 21111 tablet_service.cc:1514] Processing DeleteTablet for tablet cd49f391d00a4f6683ca2972e8ee5371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:36418
I20260619 10:21:35.390215 21359 tablet_replica.cc:330] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: stopping tablet replica
I20260619 10:21:35.390517 21359 raft_consensus.cc:2229] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:35.390599 21361 tablet_replica.cc:330] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: stopping tablet replica
I20260619 10:21:35.390426 21360 tablet_replica.cc:330] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: stopping tablet replica
I20260619 10:21:35.390750 21359 raft_consensus.cc:2258] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.390758 21360 raft_consensus.cc:2229] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:35.390851 21360 raft_consensus.cc:2258] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.391228 21359 ts_tablet_manager.cc:1905] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.391583 21251 tablet_service.cc:1467] Processing CreateTablet for tablet 7284af590a6649d5b152d477275e85b1 (DEFAULT_TABLE table=test-table [id=82d9cfb68efc4b88aa936701012ed19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.391692 21251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284af590a6649d5b152d477275e85b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.391891 21111 tablet_service.cc:1467] Processing CreateTablet for tablet 7284af590a6649d5b152d477275e85b1 (DEFAULT_TABLE table=test-table [id=82d9cfb68efc4b88aa936701012ed19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.392017 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284af590a6649d5b152d477275e85b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.392841 21338 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Bootstrap starting.
I20260619 10:21:35.393224 21339 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Bootstrap starting.
I20260619 10:21:35.393246 21338 tablet_bootstrap.cc:654] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.393316 20971 tablet_service.cc:1467] Processing CreateTablet for tablet 7284af590a6649d5b152d477275e85b1 (DEFAULT_TABLE table=test-table [id=82d9cfb68efc4b88aa936701012ed19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:35.393419 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284af590a6649d5b152d477275e85b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.393431 21359 ts_tablet_manager.cc:1918] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:35.393579 21359 log.cc:1196] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/cd49f391d00a4f6683ca2972e8ee5371
I20260619 10:21:35.393632 21339 tablet_bootstrap.cc:654] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.393915 21359 ts_tablet_manager.cc:1939] T cd49f391d00a4f6683ca2972e8ee5371 P ed1863a798ad46e3acb6777583ce5dc2: Deleting consensus metadata
I20260619 10:21:35.394443 20819 catalog_manager.cc:4841] TS ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095): tablet cd49f391d00a4f6683ca2972e8ee5371 (table test-table [id=fa20807f9e644e1e87a1cacc876ba598]) successfully deleted
I20260619 10:21:35.394750 21337 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Bootstrap starting.
I20260619 10:21:35.395015 21361 raft_consensus.cc:2229] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:35.395155 21361 raft_consensus.cc:2258] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.395165 21337 tablet_bootstrap.cc:654] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.395598 21360 ts_tablet_manager.cc:1905] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.395828 21361 ts_tablet_manager.cc:1905] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.396965 21361 ts_tablet_manager.cc:1918] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:35.397034 21361 log.cc:1196] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/cd49f391d00a4f6683ca2972e8ee5371
I20260619 10:21:35.397285 21361 ts_tablet_manager.cc:1939] T cd49f391d00a4f6683ca2972e8ee5371 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting consensus metadata
I20260619 10:21:35.395798 21337 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: No bootstrap required, opened a new log
I20260619 10:21:35.397639 21337 ts_tablet_manager.cc:1397] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 10:21:35.397686 20820 catalog_manager.cc:4841] TS 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297): tablet cd49f391d00a4f6683ca2972e8ee5371 (table test-table [id=fa20807f9e644e1e87a1cacc876ba598]) successfully deleted
I20260619 10:21:35.397784 21337 raft_consensus.cc:348] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.397864 21337 raft_consensus.cc:374] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.397889 21337 raft_consensus.cc:729] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e28b203843431cb21542ed02474534, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.397927 21337 consensus_queue.cc:260] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.398007 21337 ts_tablet_manager.cc:1428] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.398954 21360 ts_tablet_manager.cc:1918] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:35.399034 21360 log.cc:1196] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/cd49f391d00a4f6683ca2972e8ee5371
I20260619 10:21:35.399274 21360 ts_tablet_manager.cc:1939] T cd49f391d00a4f6683ca2972e8ee5371 P 89e28b203843431cb21542ed02474534: Deleting consensus metadata
I20260619 10:21:35.399637 20820 catalog_manager.cc:4841] TS 89e28b203843431cb21542ed02474534 (127.19.65.129:37223): tablet cd49f391d00a4f6683ca2972e8ee5371 (table test-table [id=fa20807f9e644e1e87a1cacc876ba598]) successfully deleted
I20260619 10:21:35.399668 21338 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: No bootstrap required, opened a new log
I20260619 10:21:35.399710 21338 ts_tablet_manager.cc:1397] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 10:21:35.399850 21338 raft_consensus.cc:348] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.399928 21338 raft_consensus.cc:374] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.399957 21338 raft_consensus.cc:729] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1863a798ad46e3acb6777583ce5dc2, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.400008 21338 consensus_queue.cc:260] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.400103 21338 ts_tablet_manager.cc:1428] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.403651 21339 tablet_bootstrap.cc:492] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: No bootstrap required, opened a new log
I20260619 10:21:35.403704 21339 ts_tablet_manager.cc:1397] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260619 10:21:35.403836 21339 raft_consensus.cc:348] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.403900 21339 raft_consensus.cc:374] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.403921 21339 raft_consensus.cc:729] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae6ba1914b2435bb76c46a7b85406b7, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.403964 21339 consensus_queue.cc:260] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.404045 21339 ts_tablet_manager.cc:1428] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 10:21:35.404991 21178 tablet.cc:2369] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:35.505993 21038 tablet.cc:2369] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:35.525134 21343 raft_consensus.cc:482] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:35.525241 21343 raft_consensus.cc:504] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.525444 21343 leader_election.cc:288] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297), ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095)
I20260619 10:21:35.525681 21271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284af590a6649d5b152d477275e85b1" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1863a798ad46e3acb6777583ce5dc2" is_pre_election: true
I20260619 10:21:35.525676 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284af590a6649d5b152d477275e85b1" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" is_pre_election: true
I20260619 10:21:35.525775 21271 raft_consensus.cc:2454] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 0.
I20260619 10:21:35.525789 21131 raft_consensus.cc:2454] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 0.
I20260619 10:21:35.525950 20923 leader_election.cc:302] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e28b203843431cb21542ed02474534, ed1863a798ad46e3acb6777583ce5dc2; no voters:
I20260619 10:21:35.526058 21343 raft_consensus.cc:2781] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:35.526108 21343 raft_consensus.cc:482] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:35.526161 21343 raft_consensus.cc:3037] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.526911 21343 raft_consensus.cc:504] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.527072 21343 leader_election.cc:288] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297), ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095)
I20260619 10:21:35.527205 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284af590a6649d5b152d477275e85b1" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
I20260619 10:21:35.527284 21131 raft_consensus.cc:3037] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.527266 21271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284af590a6649d5b152d477275e85b1" candidate_uuid: "89e28b203843431cb21542ed02474534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1863a798ad46e3acb6777583ce5dc2"
I20260619 10:21:35.527452 21271 raft_consensus.cc:3037] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.528013 21131 raft_consensus.cc:2454] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 1.
I20260619 10:21:35.528179 21271 raft_consensus.cc:2454] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e28b203843431cb21542ed02474534 in term 1.
I20260619 10:21:35.528200 20924 leader_election.cc:302] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae6ba1914b2435bb76c46a7b85406b7, 89e28b203843431cb21542ed02474534; no voters:
I20260619 10:21:35.528281 21343 raft_consensus.cc:2781] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:35.528339 21343 raft_consensus.cc:686] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 1 LEADER]: Becoming Leader. State: Replica: 89e28b203843431cb21542ed02474534, State: Running, Role: LEADER
I20260619 10:21:35.528383 21343 consensus_queue.cc:237] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.528955 20834 catalog_manager.cc:5495] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e28b203843431cb21542ed02474534 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "89e28b203843431cb21542ed02474534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } health_report { overall_health: UNKNOWN } } }
W20260619 10:21:35.553146 21318 tablet.cc:2369] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864494, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:35.562317 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.555915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44885 (local address 127.0.0.1:54256)
0619 10:21:35.556198 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.556208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.556270 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.556386 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.556387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.556647 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:35.556687 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.556689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.557272 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.557273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.557845 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.557847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.558157 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.558157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.558193 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.558699 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:35.558703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:35.560071 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:35.561423 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.561428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.561430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.561621 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.561786 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:35.561786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:35.561787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:35.561838 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:35.562036 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:35.562041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:35.562091 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.562141 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.562224 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":202,"threads_started":1}
I20260619 10:21:35.562356 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.556100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.19.65.190:44885)
0619 10:21:35.556174 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.556176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.556272 (+ 96us) server_negotiation.cc:407] Connection header received
0619 10:21:35.556308 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.556309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.556317 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.556370 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.556746 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.557211 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.557917 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.558095 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.558146 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.560150 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.560151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.560151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.560157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.561348 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.561684 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.561685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.561685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.561731 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.561887 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.561888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.561888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.561985 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.562032 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.562187 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.562268 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":50}
I20260619 10:21:35.564456 20834 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:54256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:21:35.564679 20834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:35.566067 20971 tablet_service.cc:1467] Processing CreateTablet for tablet c306e1e667324b87ac921787162fe4de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb03d3b617434887acf803e851f9073a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:35.566085 21251 tablet_service.cc:1467] Processing CreateTablet for tablet c306e1e667324b87ac921787162fe4de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb03d3b617434887acf803e851f9073a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:35.566217 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306e1e667324b87ac921787162fe4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.566227 21251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306e1e667324b87ac921787162fe4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.567262 21337 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: Bootstrap starting.
I20260619 10:21:35.567373 21338 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: Bootstrap starting.
I20260619 10:21:35.567618 21337 tablet_bootstrap.cc:654] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.567720 21338 tablet_bootstrap.cc:654] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.568077 21111 tablet_service.cc:1467] Processing CreateTablet for tablet c306e1e667324b87ac921787162fe4de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb03d3b617434887acf803e851f9073a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:35.568118 21337 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: No bootstrap required, opened a new log
I20260619 10:21:35.568225 21337 ts_tablet_manager.cc:1397] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:35.568240 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c306e1e667324b87ac921787162fe4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:35.568357 21337 raft_consensus.cc:348] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.568426 21337 raft_consensus.cc:374] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.568440 21337 raft_consensus.cc:729] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e28b203843431cb21542ed02474534, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.568490 21337 consensus_queue.cc:260] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.568589 21337 ts_tablet_manager.cc:1428] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.568602 21343 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.568671 21343 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.569730 21339 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: Bootstrap starting.
I20260619 10:21:35.569873 21338 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: No bootstrap required, opened a new log
I20260619 10:21:35.569947 21338 ts_tablet_manager.cc:1397] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:21:35.570113 21338 raft_consensus.cc:348] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.570152 21339 tablet_bootstrap.cc:654] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:35.570184 21338 raft_consensus.cc:374] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.570209 21338 raft_consensus.cc:729] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1863a798ad46e3acb6777583ce5dc2, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.570259 21338 consensus_queue.cc:260] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.570350 21338 ts_tablet_manager.cc:1428] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.570423 21344 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.570529 21344 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.570770 21339 tablet_bootstrap.cc:492] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: No bootstrap required, opened a new log
I20260619 10:21:35.570818 21339 ts_tablet_manager.cc:1397] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:35.570978 21339 raft_consensus.cc:348] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.571086 21339 raft_consensus.cc:374] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:35.571108 21339 raft_consensus.cc:729] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae6ba1914b2435bb76c46a7b85406b7, State: Initialized, Role: FOLLOWER
I20260619 10:21:35.571178 21339 consensus_queue.cc:260] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.571269 21339 ts_tablet_manager.cc:1428] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:35.571368 21345 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.571477 21345 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.624665 21344 raft_consensus.cc:482] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:35.624773 21344 raft_consensus.cc:504] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.625130 21344 leader_election.cc:288] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89e28b203843431cb21542ed02474534 (127.19.65.129:37223), 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297)
I20260619 10:21:35.628439 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.625301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:37223 (local address 127.19.65.131:59651)
0619 10:21:35.625379 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.625393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.625447 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.625596 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.625598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.625600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.625666 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.625668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.626304 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.626305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.627670 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.627673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.628263 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.628264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.628318 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.628321 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.628361 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.628390 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 10:21:35.628484 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.625401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59651 (local address 127.19.65.129:37223)
0619 10:21:35.625470 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.625472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.625480 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:35.625509 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.625509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.625522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.625574 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.625745 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.626224 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.627747 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.628193 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.628263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.628271 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.628383 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.628421 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:21:35.628719 20991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306e1e667324b87ac921787162fe4de" candidate_uuid: "ed1863a798ad46e3acb6777583ce5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e28b203843431cb21542ed02474534" is_pre_election: true
I20260619 10:21:35.628870 20991 raft_consensus.cc:2454] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1863a798ad46e3acb6777583ce5dc2 in term 0.
I20260619 10:21:35.628882 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.625481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:58365 (local address 127.19.65.130:36297)
0619 10:21:35.625524 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.625526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.625693 (+ 167us) server_negotiation.cc:407] Connection header received
0619 10:21:35.625752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.625753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.625763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.625818 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:35.625998 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.626486 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.628049 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.628554 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.628621 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.628629 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.628788 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.628826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:35.629072 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.625426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:36297 (local address 127.19.65.131:58365)
0619 10:21:35.625614 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.625623 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.625700 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.625828 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.625828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.625830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:35.625920 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.625922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.626600 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.626601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.627961 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.627971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.628622 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.628623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.628700 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.628703 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.628954 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.629003 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260619 10:21:35.629055 21204 leader_election.cc:302] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e28b203843431cb21542ed02474534, ed1863a798ad46e3acb6777583ce5dc2; no voters:
I20260619 10:21:35.629156 21344 raft_consensus.cc:2781] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:35.629187 21344 raft_consensus.cc:482] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:35.629217 21344 raft_consensus.cc:3037] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.629287 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306e1e667324b87ac921787162fe4de" candidate_uuid: "ed1863a798ad46e3acb6777583ce5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" is_pre_election: true
I20260619 10:21:35.629390 21131 raft_consensus.cc:2454] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1863a798ad46e3acb6777583ce5dc2 in term 0.
I20260619 10:21:35.629984 21344 raft_consensus.cc:504] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.630148 21344 leader_election.cc:288] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 89e28b203843431cb21542ed02474534 (127.19.65.129:37223), 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297)
I20260619 10:21:35.630291 20991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306e1e667324b87ac921787162fe4de" candidate_uuid: "ed1863a798ad46e3acb6777583ce5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e28b203843431cb21542ed02474534"
I20260619 10:21:35.630354 20991 raft_consensus.cc:3037] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.630301 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c306e1e667324b87ac921787162fe4de" candidate_uuid: "ed1863a798ad46e3acb6777583ce5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae6ba1914b2435bb76c46a7b85406b7"
I20260619 10:21:35.630394 21131 raft_consensus.cc:3037] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:35.630929 21131 raft_consensus.cc:2454] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1863a798ad46e3acb6777583ce5dc2 in term 1.
I20260619 10:21:35.631059 20991 raft_consensus.cc:2454] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1863a798ad46e3acb6777583ce5dc2 in term 1.
I20260619 10:21:35.631083 21204 leader_election.cc:302] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae6ba1914b2435bb76c46a7b85406b7, ed1863a798ad46e3acb6777583ce5dc2; no voters:
I20260619 10:21:35.631155 21344 raft_consensus.cc:2781] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:35.631278 21344 raft_consensus.cc:686] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [term 1 LEADER]: Becoming Leader. State: Replica: ed1863a798ad46e3acb6777583ce5dc2, State: Running, Role: LEADER
I20260619 10:21:35.631371 21344 consensus_queue.cc:237] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } }
I20260619 10:21:35.631703 21376 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: TxnStatusTablet state changed. Reason: New leader ed1863a798ad46e3acb6777583ce5dc2. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.631798 21376 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:35.631995 21378 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:21:35.632004 20833 catalog_manager.cc:5495] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } health_report { overall_health: HEALTHY } } }
I20260619 10:21:35.632112 21378 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:21:35.646310 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.643916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.19.65.131:45095)
0619 10:21:35.643972 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.643974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.643985 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:35.644070 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.644072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.644117 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.644198 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:35.644303 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.644769 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.645528 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.645727 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.645801 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.645986 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.646082 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.646172 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.646224 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.646242 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:21:35.646332 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.643856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45095 (local address 127.0.0.1:59288)
0619 10:21:35.643924 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.643952 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.644006 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.644203 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.644204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.644205 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:35.644251 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.644253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.644856 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.644856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.645423 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.645425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.645800 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.645801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.645867 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.645883 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.646144 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.646164 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.646219 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.646264 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 10:21:35.647327 21131 raft_consensus.cc:1261] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Refusing update from remote peer ed1863a798ad46e3acb6777583ce5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.647430 20991 raft_consensus.cc:1261] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Refusing update from remote peer ed1863a798ad46e3acb6777583ce5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.647584 21344 consensus_queue.cc:1035] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.647684 21376 consensus_queue.cc:1035] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.648334 21345 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: TxnStatusTablet state changed. Reason: New leader ed1863a798ad46e3acb6777583ce5dc2. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.648452 21345 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.648428 21343 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: TxnStatusTablet state changed. Reason: New leader ed1863a798ad46e3acb6777583ce5dc2. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.648509 21343 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.649044 21344 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.649135 21344 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:35.649312 21344 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.649386 21344 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P ed1863a798ad46e3acb6777583ce5dc2: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:35.649647 21350 mvcc.cc:205] Tried to move back new op lower bound from 7298516974169411584 to 7298516974106607616. Current Snapshot: MvccSnapshot[applied={T|T < 7298516974169411584}]
I20260619 10:21:35.649824 21343 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.649899 21343 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 89e28b203843431cb21542ed02474534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.650358 21345 tablet_replica.cc:439] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed1863a798ad46e3acb6777583ce5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e28b203843431cb21542ed02474534" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 37223 } } peers { permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 } } peers { permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 } } }
I20260619 10:21:35.650446 21345 tablet_replica.cc:442] T c306e1e667324b87ac921787162fe4de P 6ae6ba1914b2435bb76c46a7b85406b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:35.654924 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.652716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45245 (local address 127.19.65.131:45095)
0619 10:21:35.652747 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.652749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.652875 (+ 126us) server_negotiation.cc:407] Connection header received
0619 10:21:35.652937 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.652938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.652948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.652999 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:35.653093 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.653507 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.654234 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.654411 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.654500 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.654679 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.654711 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.654767 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.654852 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.654867 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:21:35.654981 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.652662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45095 (local address 127.19.65.129:45245)
0619 10:21:35.652813 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.652822 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.652890 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.653002 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.653003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.653004 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:35.653048 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.653049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.653599 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.653600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.654156 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.654157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.654512 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.654513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.654575 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.654587 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.654791 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.654793 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.654847 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.654893 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260619 10:21:35.656742 21271 raft_consensus.cc:1261] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Refusing update from remote peer 89e28b203843431cb21542ed02474534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.656771 21131 raft_consensus.cc:1261] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Refusing update from remote peer 89e28b203843431cb21542ed02474534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:35.656935 21343 consensus_queue.cc:1035] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1863a798ad46e3acb6777583ce5dc2" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.657092 21347 consensus_queue.cc:1035] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae6ba1914b2435bb76c46a7b85406b7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:35.664307 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.661970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59431 (local address 127.19.65.129:37223)
0619 10:21:35.662005 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.662007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.662015 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:35.662034 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.662035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.662042 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.662101 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:35.662241 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.662690 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.663511 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.663779 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.663845 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.664040 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.664075 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.664121 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.664208 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.664227 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:35.664309 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.661866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:37223 (local address 127.19.65.131:59431)
0619 10:21:35.661930 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:35.661941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:35.661980 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:35.662108 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:35.662108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:35.662110 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:35.662170 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.662171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.662777 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.662778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.663410 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:35.663412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.663846 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.663847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:35.663953 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.663969 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:35.664144 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:35.664146 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:35.664189 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:21:35.664229 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 10:21:35.672937 20833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:35.673043 20833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:35.673622 20833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e56c5693a604d18b451072d71f82a28: Sending DeleteTablet for 3 replicas of tablet 7284af590a6649d5b152d477275e85b1
I20260619 10:21:35.673931 21111 tablet_service.cc:1514] Processing DeleteTablet for tablet 7284af590a6649d5b152d477275e85b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:36418
I20260619 10:21:35.673962 21251 tablet_service.cc:1514] Processing DeleteTablet for tablet 7284af590a6649d5b152d477275e85b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59282
I20260619 10:21:35.674420 21361 tablet_replica.cc:330] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: stopping tablet replica
I20260619 10:21:35.674487 21361 raft_consensus.cc:2229] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:35.674528 21361 raft_consensus.cc:2258] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.674592 21359 tablet_replica.cc:330] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: stopping tablet replica
I20260619 10:21:35.674641 21359 raft_consensus.cc:2229] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:35.674664 21359 raft_consensus.cc:2258] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.674757 20971 tablet_service.cc:1514] Processing DeleteTablet for tablet 7284af590a6649d5b152d477275e85b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:35 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35334
I20260619 10:21:35.674871 21360 tablet_replica.cc:330] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: stopping tablet replica
I20260619 10:21:35.674919 21360 raft_consensus.cc:2229] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:35.674986 21360 raft_consensus.cc:2258] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:35.675046 21359 ts_tablet_manager.cc:1905] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.675199 21361 ts_tablet_manager.cc:1905] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.675431 21360 ts_tablet_manager.cc:1905] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:35.676739 21359 ts_tablet_manager.cc:1918] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:35.676741 21361 ts_tablet_manager.cc:1918] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:35.676787 21359 log.cc:1196] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/7284af590a6649d5b152d477275e85b1
I20260619 10:21:35.676795 21361 log.cc:1196] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/7284af590a6649d5b152d477275e85b1
I20260619 10:21:35.676852 21360 ts_tablet_manager.cc:1918] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:35.676898 21360 log.cc:1196] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/7284af590a6649d5b152d477275e85b1
I20260619 10:21:35.677016 21361 ts_tablet_manager.cc:1939] T 7284af590a6649d5b152d477275e85b1 P 6ae6ba1914b2435bb76c46a7b85406b7: Deleting consensus metadata
I20260619 10:21:35.677026 21359 ts_tablet_manager.cc:1939] T 7284af590a6649d5b152d477275e85b1 P ed1863a798ad46e3acb6777583ce5dc2: Deleting consensus metadata
I20260619 10:21:35.677110 21360 ts_tablet_manager.cc:1939] T 7284af590a6649d5b152d477275e85b1 P 89e28b203843431cb21542ed02474534: Deleting consensus metadata
I20260619 10:21:35.677294 20819 catalog_manager.cc:4841] TS ed1863a798ad46e3acb6777583ce5dc2 (127.19.65.131:45095): tablet 7284af590a6649d5b152d477275e85b1 (table test-table [id=82d9cfb68efc4b88aa936701012ed19e]) successfully deleted
I20260619 10:21:35.677350 20820 catalog_manager.cc:4841] TS 6ae6ba1914b2435bb76c46a7b85406b7 (127.19.65.130:36297): tablet 7284af590a6649d5b152d477275e85b1 (table test-table [id=82d9cfb68efc4b88aa936701012ed19e]) successfully deleted
I20260619 10:21:35.677395 20820 catalog_manager.cc:4841] TS 89e28b203843431cb21542ed02474534 (127.19.65.129:37223): tablet 7284af590a6649d5b152d477275e85b1 (table test-table [id=82d9cfb68efc4b88aa936701012ed19e]) successfully deleted
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:35.685539 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.674505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.19.65.129:37223)
0619 10:21:35.674540 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.674542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.677546 (+ 3004us) server_negotiation.cc:407] Connection header received
0619 10:21:35.677594 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.677595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.677625 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.677687 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.678237 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.678991 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.679714 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.680002 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.680068 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.682736 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.682743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.682743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.682750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.684475 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.684806 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.684809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.684810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.684873 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.685062 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.685063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.685064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.685214 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.685342 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.685379 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.685451 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 10:21:35.685767 21026 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35344
I20260619 10:21:35.692152 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:35.686658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.19.65.190:44885)
0619 10:21:35.686702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:35.686704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:35.686712 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:35.686774 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:35.686775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:35.686784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:35.686833 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:35.687277 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.687759 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.688423 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:35.688607 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:35.688673 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:35.689682 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:35.689683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:35.689683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:35.689690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:35.690893 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.691222 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.691223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.691223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.691280 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:35.691500 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:35.691500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:35.691501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:35.691671 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:35.691764 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:35.691955 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 10:21:35.692045 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 10:21:35.692458 20833 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272
I20260619 10:21:35.692974 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20906
I20260619 10:21:35.699796 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21046
I20260619 10:21:35.704291 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21186
I20260619 10:21:35.709404 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 20802
2026-06-19T10:21:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2161 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[21398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[21398](info): set up 2 sockets
Jun 19 10:21:35 dist-test-slave-xd2q krb5kdc[21398](info): commencing operation
krb5kdc: starting...
W20260619 10:21:36.726809 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:36 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:36.838691 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39915
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:36.913800 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:36.914007 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:36.914038 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:36.915143 21411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:36.915205 21411 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:21:36.915225 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:36.915246 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:36.915275 21411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:36.915331 21411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:36.916755 21411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39915
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:36.917017 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:36.917224 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:36.921862 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:36.921871 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:36.921871 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:36.922083 21411 server_base.cc:1034] running on GCE node
I20260619 10:21:36.922307 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:36.922574 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:36.923746 21411 hybrid_clock.cc:648] HybridClock initialized: now 1781864496923733 us; error 29 us; skew 500 ppm
Jun 19 10:21:36 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864496, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:36.926070 21411 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:36.926503 21411 webserver.cc:460] Webserver started at http://127.19.65.190:38241/ using document root <none> and password file <none>
I20260619 10:21:36.926716 21411 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:36.926781 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:36.926925 21411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:36.927942 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "a51da65e82aa410f9f849e8505409682"
format_stamp: "Formatted at 2026-06-19 10:21:36 on dist-test-slave-xd2q"
server_key: "936d7417d4736c177b340328ce6f6b37"
server_key_iv: "11d9ef1a9e81a0303c2dc90e4eee1741"
server_key_version: "encryptionkey@0"
I20260619 10:21:36.928285 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "a51da65e82aa410f9f849e8505409682"
format_stamp: "Formatted at 2026-06-19 10:21:36 on dist-test-slave-xd2q"
server_key: "936d7417d4736c177b340328ce6f6b37"
server_key_iv: "11d9ef1a9e81a0303c2dc90e4eee1741"
server_key_version: "encryptionkey@0"
I20260619 10:21:36.929534 21411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:36.930179 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:36.930354 21411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:36.930445 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "a51da65e82aa410f9f849e8505409682"
format_stamp: "Formatted at 2026-06-19 10:21:36 on dist-test-slave-xd2q"
server_key: "936d7417d4736c177b340328ce6f6b37"
server_key_iv: "11d9ef1a9e81a0303c2dc90e4eee1741"
server_key_version: "encryptionkey@0"
I20260619 10:21:36.930522 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:36.957952 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:36.960371 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:36.960534 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:36.965548 21411 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39915
I20260619 10:21:36.965595 21488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:39915 every 8 connection(s)
I20260619 10:21:36.965953 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:36.966593 21489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:36.968782 21489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Bootstrap starting.
I20260619 10:21:36.969364 21489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:36.969672 21489 log.cc:824] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:36.970323 21489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: No bootstrap required, opened a new log
I20260619 10:21:36.972402 21489 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } }
I20260619 10:21:36.972543 21489 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:36.972564 21489 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51da65e82aa410f9f849e8505409682, State: Initialized, Role: FOLLOWER
I20260619 10:21:36.972700 21489 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } }
I20260619 10:21:36.972721 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21411
I20260619 10:21:36.972780 21489 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:36.972801 21489 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:36.972827 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:36.972859 21489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:36.972988 19718 external_mini_cluster.cc:1428] Setting key b9475e3dfe59463d511e2902e445411d
I20260619 10:21:36.973613 21489 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } }
I20260619 10:21:36.973739 21489 leader_election.cc:302] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51da65e82aa410f9f849e8505409682; no voters:
I20260619 10:21:36.973881 21489 leader_election.cc:288] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:36.973912 21494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:36.974121 21489 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:36.974751 21494 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [term 1 LEADER]: Becoming Leader. State: Replica: a51da65e82aa410f9f849e8505409682, State: Running, Role: LEADER
I20260619 10:21:36.974849 21494 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } }
I20260619 10:21:36.975240 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51da65e82aa410f9f849e8505409682. Latest consensus state: current_term: 1 leader_uuid: "a51da65e82aa410f9f849e8505409682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } } }
I20260619 10:21:36.975368 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:36.975503 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51da65e82aa410f9f849e8505409682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51da65e82aa410f9f849e8505409682" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39915 } } }
I20260619 10:21:36.975561 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682 [sys.catalog]: This master's current role is: LEADER
W20260619 10:21:36.976071 21509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:36.976492 21509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:36.976559 21507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:36.976698 21507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:36.978140 21507 catalog_manager.cc:1347] Generated new cluster ID: 927c21d44505447a8b0acf23eb43ef24
I20260619 10:21:36.978190 21507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:36 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:36.983335 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:36.973485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.19.65.190:39915)
0619 10:21:36.973671 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:36.973681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:36.973734 (+ 53us) server_negotiation.cc:407] Connection header received
0619 10:21:36.973852 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:36.973858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:36.973930 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:36.974073 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:36.974672 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:36.976230 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:36.977161 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:36.977756 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:36.977860 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:36.980175 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:36.980179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:36.980179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:36.980189 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:36.981790 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:36.982191 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:36.982194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:36.982194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:36.982283 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:36.982499 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:36.982500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:36.982500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:36.982709 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:36.982880 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:36.982990 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 10:21:36.983125 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260619 10:21:37.005811 21507 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:37.006546 21507 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:37.022922 21507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Generated new TSK 0
I20260619 10:21:37.023142 21507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:37.055734 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39915
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:37.131778 21515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:37.131961 21515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:37.131982 21515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:37.133162 21515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:37.133216 21515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:37.133276 21515 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:21:37.133308 21515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:37.133332 21515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:37.134752 21515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:37.134912 21515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:37.135113 21515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:37.139667 21521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.139660 21523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.139669 21520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:37.139943 21515 server_base.cc:1034] running on GCE node
I20260619 10:21:37.140129 21515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:37.140350 21515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:37.141490 21515 hybrid_clock.cc:648] HybridClock initialized: now 1781864497141481 us; error 31 us; skew 500 ppm
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:37.143968 21515 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:37.144358 21515 webserver.cc:460] Webserver started at http://127.19.65.129:46335/ using document root <none> and password file <none>
I20260619 10:21:37.144565 21515 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:37.144606 21515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:37.144744 21515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:37.145705 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "3d5de23b097b40b5a860631b5cfb8926"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "b65ab4d27c06bdd1d5a93c89a38b5999"
server_key_iv: "090a6d50f248f2941714c47302857154"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.146039 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "3d5de23b097b40b5a860631b5cfb8926"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "b65ab4d27c06bdd1d5a93c89a38b5999"
server_key_iv: "090a6d50f248f2941714c47302857154"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.147260 21515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:37.148001 21530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.148176 21515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:37.148269 21515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "3d5de23b097b40b5a860631b5cfb8926"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "b65ab4d27c06bdd1d5a93c89a38b5999"
server_key_iv: "090a6d50f248f2941714c47302857154"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.148350 21515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:37.159672 21515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:37.161747 21515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:37.161895 21515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:37.162336 21515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:37.162398 21515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.162442 21515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:37.162482 21515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.169132 21515 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:38111
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:37.169162 21645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:38111 every 8 connection(s)
I20260619 10:21:37.169507 21515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:37.169824 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21515
I20260619 10:21:37.169952 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:37.170133 19718 external_mini_cluster.cc:1428] Setting key 9c709ef8562c97fbff8316a389a173b3
I20260619 10:21:37.173534 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.162913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:35075 (local address 127.19.65.190:39915)
0619 10:21:37.162965 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.162967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.163145 (+ 178us) server_negotiation.cc:407] Connection header received
0619 10:21:37.163633 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.163634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.163651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.163715 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.164469 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.165302 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.166836 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.167201 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.167279 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.169706 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.169708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.169708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.169717 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.172064 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.172561 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.172562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.172563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.172611 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.172876 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.172876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.172876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.172978 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.173025 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.173330 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.173402 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:21:37.173764 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.162846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.129:35075)
0619 10:21:37.163013 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.163060 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.163528 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.163777 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.163780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.164108 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.164388 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.164393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.165388 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.165389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.166739 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.166742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.167320 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.167322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.167440 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.168063 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.168071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.169611 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.172167 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.172174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.172176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.172485 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.172680 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.172680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.172681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.172815 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.173046 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.173052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.173203 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.173312 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.173490 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260619 10:21:37.175118 21646 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39915
I20260619 10:21:37.175339 21646 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:37.175508 21646 heartbeater.cc:511] Master 127.19.65.190:39915 requested a full tablet report, sending...
I20260619 10:21:37.175976 21442 ts_manager.cc:194] Registered new tserver with Master: 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111)
I20260619 10:21:37.176882 21442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:35075
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:21:37.181305 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.175239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:34713 (local address 127.19.65.190:39915)
0619 10:21:37.175275 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.175277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.175413 (+ 136us) server_negotiation.cc:407] Connection header received
0619 10:21:37.175531 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.175532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.175558 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.175590 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.176044 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.176808 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.177941 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.178122 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.178177 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.179165 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.179166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.179166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.179172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.180399 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.180678 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.180678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.180679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.180720 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.180860 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.180860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.180860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.180946 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.180990 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.181146 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.181224 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":19}
I20260619 10:21:37.181313 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.175158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.129:34713)
0619 10:21:37.175344 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.175356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.175440 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.175671 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.175671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.175888 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.175960 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.175962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.176914 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.176915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.177832 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.177835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.178202 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.178203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.178280 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.178662 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.178668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.179093 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.180472 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.180473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.180474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.180623 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.180770 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.180771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.180771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.180817 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.180993 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.180993 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.181035 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.181118 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.181212 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:37.195400 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39915
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:37.272382 21655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:37.272557 21655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:37.272579 21655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:37.273620 21655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:37.273677 21655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:37.273713 21655 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:21:37.273743 21655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:37.273777 21655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:37.275319 21655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:37.275542 21655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:37.275755 21655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:37.280471 21660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.280476 21663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.280476 21661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:37.280709 21655 server_base.cc:1034] running on GCE node
I20260619 10:21:37.280894 21655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:37.281149 21655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:37.282322 21655 hybrid_clock.cc:648] HybridClock initialized: now 1781864497282322 us; error 63 us; skew 500 ppm
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:37.284754 21655 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:37.285207 21655 webserver.cc:460] Webserver started at http://127.19.65.130:46241/ using document root <none> and password file <none>
I20260619 10:21:37.285434 21655 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:37.285497 21655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:37.285645 21655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:37.286718 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "9930d2eac87b4ed197dcc42a425b7374"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "dbcc68c196e0bcecb415f29eb5a8d4a9"
server_key_iv: "ac679a1353c16296023835eb2cabffa0"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.287065 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "9930d2eac87b4ed197dcc42a425b7374"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "dbcc68c196e0bcecb415f29eb5a8d4a9"
server_key_iv: "ac679a1353c16296023835eb2cabffa0"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.288409 21655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:37.289143 21670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.289338 21655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:37.289450 21655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "9930d2eac87b4ed197dcc42a425b7374"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "dbcc68c196e0bcecb415f29eb5a8d4a9"
server_key_iv: "ac679a1353c16296023835eb2cabffa0"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.289534 21655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:37.301043 21655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:37.302772 21655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:37.302927 21655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:37.303488 21655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:37.303546 21655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.303591 21655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:37.303625 21655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.310566 21655 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45017
I20260619 10:21:37.310680 21785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:45017 every 8 connection(s)
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:37.310940 21655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:37.313773 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.303914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33147 (local address 127.19.65.190:39915)
0619 10:21:37.303985 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.303987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.304220 (+ 233us) server_negotiation.cc:407] Connection header received
0619 10:21:37.304899 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.304900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.304910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.305015 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.305862 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.306682 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.308305 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.308641 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.308726 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.311385 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.311386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.311387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.311394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.312793 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.313125 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.313126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.313126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.313172 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.313317 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.313318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.313318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.313402 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.313448 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.313656 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.313715 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:21:37.313920 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.303888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.130:33147)
0619 10:21:37.304068 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.304119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.304786 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.305086 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.305091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.305468 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.305755 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.305761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.306776 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.306778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.308217 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.308220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.308726 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.308727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.308887 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.309380 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.309390 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.311257 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.312860 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.312864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.312866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.313065 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.313211 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.313211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.313211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.313284 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.313456 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.313460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.313555 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.313630 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.313734 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260619 10:21:37.314361 21786 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39915
I20260619 10:21:37.314457 21786 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:37.314620 21786 heartbeater.cc:511] Master 127.19.65.190:39915 requested a full tablet report, sending...
I20260619 10:21:37.314996 21443 ts_manager.cc:194] Registered new tserver with Master: 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017)
I20260619 10:21:37.315505 21443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:33147
I20260619 10:21:37.319500 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21655
I20260619 10:21:37.319608 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:37.319804 19718 external_mini_cluster.cc:1428] Setting key f1e642ebbcca96c69e3fd8b49f82fe83
I20260619 10:21:37.320838 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.315377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:38961 (local address 127.19.65.190:39915)
0619 10:21:37.315415 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.315417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.315539 (+ 122us) server_negotiation.cc:407] Connection header received
0619 10:21:37.315565 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.315566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.315576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.315635 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.316031 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.316550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.317164 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.317332 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.317378 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.318163 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.318163 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.318164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.318169 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.319686 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.320049 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.320050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.320050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.320109 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.320305 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.320306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.320306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.320454 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.320529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.320685 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.320754 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:37.320855 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.315351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.130:38961)
0619 10:21:37.315464 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.315474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.315531 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.315715 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.315715 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.315902 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.315953 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.315954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.316631 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.316632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.317101 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.317103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.317391 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.317391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.317448 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.317837 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.317840 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.318090 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.319761 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.319761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.319762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.319971 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.320174 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.320175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.320175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.320233 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.320533 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.320534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.320576 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.320661 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.320756 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:37.342063 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39915
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:37.419720 21795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:37.419895 21795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:37.419919 21795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:37.421025 21795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:37.421066 21795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:37.421132 21795 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:21:37.421162 21795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:21:37.421190 21795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:37.422577 21795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:37.422799 21795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:37.423005 21795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:21:37.427641 21795 server_base.cc:1034] running on GCE node
W20260619 10:21:37.427613 21801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.427670 21803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:37.427613 21800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:37.427950 21795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:37.428181 21795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:37.429378 21795 hybrid_clock.cc:648] HybridClock initialized: now 1781864497429356 us; error 30 us; skew 500 ppm
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:37.431716 21795 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:37.432063 21795 webserver.cc:460] Webserver started at http://127.19.65.131:44589/ using document root <none> and password file <none>
I20260619 10:21:37.432233 21795 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:37.432271 21795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:37.432350 21795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:37.433357 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "fd60a7b7eec345b6addd9117e70c074a"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "6bbaf65fd2400987263b3b54327e2967"
server_key_iv: "02495300b0c6392c68128fc47c082f18"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.433665 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "fd60a7b7eec345b6addd9117e70c074a"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "6bbaf65fd2400987263b3b54327e2967"
server_key_iv: "02495300b0c6392c68128fc47c082f18"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.434878 21795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:37.435608 21810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.435760 21795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:37.435840 21795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "fd60a7b7eec345b6addd9117e70c074a"
format_stamp: "Formatted at 2026-06-19 10:21:37 on dist-test-slave-xd2q"
server_key: "6bbaf65fd2400987263b3b54327e2967"
server_key_iv: "02495300b0c6392c68128fc47c082f18"
server_key_version: "encryptionkey@0"
I20260619 10:21:37.435894 21795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:37.441797 21795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:37.443444 21795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:37.443542 21795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:37.443931 21795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:37.443964 21795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.444011 21795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:37.444041 21795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.450191 21795 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45129
I20260619 10:21:37.450248 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45129 every 8 connection(s)
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:37.450538 21795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:37.453574 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.444401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43581 (local address 127.19.65.190:39915)
0619 10:21:37.444470 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.444472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.444646 (+ 174us) server_negotiation.cc:407] Connection header received
0619 10:21:37.445078 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.445079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.445092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.445176 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.445936 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.446755 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.448305 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.448629 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.448710 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.451067 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.451068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.451069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.451077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.452481 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.452821 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.452822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.452822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.452868 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.453046 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.453047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.453047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.453141 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.453199 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.453455 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.453514 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:21:37.453680 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.444341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.131:43581)
0619 10:21:37.444528 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.444566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.444989 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.445219 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.445222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.445569 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.445828 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.445835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.446869 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.446870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.448212 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.448215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.448717 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.448718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.448810 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.449292 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.449302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.450972 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.452544 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.452548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.452550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.452759 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.452923 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.452923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.452923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.453000 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.453205 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.453209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.453325 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.453402 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.453510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260619 10:21:37.454145 21926 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39915
I20260619 10:21:37.454233 21926 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:37.454388 21926 heartbeater.cc:511] Master 127.19.65.190:39915 requested a full tablet report, sending...
I20260619 10:21:37.454746 21443 ts_manager.cc:194] Registered new tserver with Master: fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:37.455265 21443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:43581
I20260619 10:21:37.455922 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 21795
I20260619 10:21:37.456023 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:37.456223 19718 external_mini_cluster.cc:1428] Setting key 4190dc75f86a23ad0c11117e1854034d
I20260619 10:21:37.457106 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:21:37.460846 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.455111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.19.65.131:48329)
0619 10:21:37.455228 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.455239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.455310 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.455465 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.455465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.455634 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.455683 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.455684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.456348 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.456348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.457111 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.457113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.457497 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.457498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.457563 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.457994 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.457998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.458242 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.459959 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.459960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.459960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.460117 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.460311 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.460312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.460312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.460361 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.460624 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.460624 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.460658 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.460712 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.460780 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260619 10:21:37.460954 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.455181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48329 (local address 127.19.65.190:39915)
0619 10:21:37.455223 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.455224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.455281 (+ 57us) server_negotiation.cc:407] Connection header received
0619 10:21:37.455368 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.455368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.455377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.455425 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.455774 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.456257 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.457191 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.457406 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.457493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.458330 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.458332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.458332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.458339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.459858 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.460188 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.460189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.460189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.460247 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.460427 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.460428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.460428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.460560 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.460624 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.460753 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.460832 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:37.479861 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.473076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.19.65.190:39915)
0619 10:21:37.473129 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.473131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.473165 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:21:37.473183 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.473184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.473191 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.473229 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.473797 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.474318 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.475194 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.475469 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.475522 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.477663 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.477664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.477665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.477670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.478843 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.479228 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.479229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.479229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.479272 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.479486 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.479487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.479487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.479588 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.479661 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.479756 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.479806 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:21:37.481431 21443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:37.482857 21443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:37.488524 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.484907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38111 (local address 127.0.0.1:52646)
0619 10:21:37.485078 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.485116 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.485175 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.485449 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.485452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.485453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.485509 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.485511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.486467 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.486468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.487344 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.487346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.488230 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.488231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.488315 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.488329 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.488408 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.488456 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260619 10:21:37.488565 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.485111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.19.65.129:38111)
0619 10:21:37.485257 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.485262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.485273 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:37.485314 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.485316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.485333 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.485438 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.485572 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.486385 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.487423 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.488140 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.488230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.488288 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.488433 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.488473 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260619 10:21:37.489264 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.484908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:45017 (local address 127.0.0.1:33576)
0619 10:21:37.485270 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.485281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.485332 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.485604 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.485605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.485607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.485656 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.485658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.486925 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.486925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.488065 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.488068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.489060 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.489061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.489102 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.489104 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.489166 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.489203 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":33,"thread_start_us":84,"threads_started":1}
I20260619 10:21:37.489411 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.485114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45129 (local address 127.0.0.1:56254)
0619 10:21:37.485278 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.485287 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.485327 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.485870 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.485871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.485873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.485904 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.485905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.487140 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.487141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.488109 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.488112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.489198 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.489200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.489257 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.489261 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.489291 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.489327 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":35,"threads_started":1}
I20260619 10:21:37.489408 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.485042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.19.65.130:45017)
0619 10:21:37.485440 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.485444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.485453 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:37.485489 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.485493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.485512 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.485593 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.485743 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.486833 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.488172 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.488919 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.488963 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.489013 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.489196 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.489230 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":81,"threads_started":1}
I20260619 10:21:37.489341 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 (DEFAULT_TABLE table=test-table [id=bce2ff7c573a4dea840530fba35d0307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.489682 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c7a4b5bacb45b98de7e31c6980e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.489727 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.485263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.19.65.131:45129)
0619 10:21:37.485709 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.485713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.485722 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:37.485755 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.485756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.485771 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.485982 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.486033 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.487061 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.488198 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.489090 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.489509 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.489566 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.489592 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.489633 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":85,"threads_started":1}
I20260619 10:21:37.490037 21720 tablet_service.cc:1467] Processing CreateTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 (DEFAULT_TABLE table=test-table [id=bce2ff7c573a4dea840530fba35d0307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.490283 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c7a4b5bacb45b98de7e31c6980e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.492488 21946 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Bootstrap starting.
I20260619 10:21:37.493232 21946 tablet_bootstrap.cc:654] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.493355 21860 tablet_service.cc:1467] Processing CreateTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 (DEFAULT_TABLE table=test-table [id=bce2ff7c573a4dea840530fba35d0307]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.493595 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c7a4b5bacb45b98de7e31c6980e347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.493577 21946 log.cc:824] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:37.493924 21947 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Bootstrap starting.
I20260619 10:21:37.494271 21946 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: No bootstrap required, opened a new log
I20260619 10:21:37.494321 21946 ts_tablet_manager.cc:1397] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:37.495999 21947 tablet_bootstrap.cc:654] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.495924 21946 raft_consensus.cc:348] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.496083 21946 raft_consensus.cc:374] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.496129 21946 raft_consensus.cc:729] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5de23b097b40b5a860631b5cfb8926, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.496246 21946 consensus_queue.cc:260] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.496032 21949 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Bootstrap starting.
I20260619 10:21:37.496340 21947 log.cc:824] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:37.496551 21946 ts_tablet_manager.cc:1428] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:21:37.496670 21646 heartbeater.cc:503] Master 127.19.65.190:39915 was elected leader, sending a full tablet report...
I20260619 10:21:37.497187 21947 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: No bootstrap required, opened a new log
I20260619 10:21:37.497246 21947 ts_tablet_manager.cc:1397] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:21:37.497260 21949 tablet_bootstrap.cc:654] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.497658 21949 log.cc:824] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:37.498392 21949 tablet_bootstrap.cc:492] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: No bootstrap required, opened a new log
I20260619 10:21:37.498456 21949 ts_tablet_manager.cc:1397] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:37.499047 21947 raft_consensus.cc:348] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.499197 21947 raft_consensus.cc:374] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.499238 21947 raft_consensus.cc:729] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.499361 21947 consensus_queue.cc:260] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.499624 21947 ts_tablet_manager.cc:1428] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:21:37.499631 21786 heartbeater.cc:503] Master 127.19.65.190:39915 was elected leader, sending a full tablet report...
I20260619 10:21:37.499963 21949 raft_consensus.cc:348] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.500079 21949 raft_consensus.cc:374] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.500133 21949 raft_consensus.cc:729] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.500236 21949 consensus_queue.cc:260] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.500550 21949 ts_tablet_manager.cc:1428] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:37.500670 21926 heartbeater.cc:503] Master 127.19.65.190:39915 was elected leader, sending a full tablet report...
I20260619 10:21:37.538635 21953 raft_consensus.cc:482] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:37.538766 21953 raft_consensus.cc:504] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.539100 21953 leader_election.cc:288] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:37.541795 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.539310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38111 (local address 127.19.65.130:52415)
0619 10:21:37.539354 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.539367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.539462 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.539610 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.539611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.539613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.539679 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.539681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.540302 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.540303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.541006 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.541008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.541535 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.541611 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.541615 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.541693 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.541730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 10:21:37.541795 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.539378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52415 (local address 127.19.65.129:38111)
0619 10:21:37.539458 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.539460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.539471 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:37.539525 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.539526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.539537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.539600 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.539759 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.540226 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541105 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.541454 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541538 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.541546 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.541701 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.541735 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 10:21:37.542017 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.539531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50965 (local address 127.19.65.131:45129)
0619 10:21:37.539569 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.539571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.539646 (+ 75us) server_negotiation.cc:407] Connection header received
0619 10:21:37.539724 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.539725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.539734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.539804 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.539947 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.540388 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541221 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.541559 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541635 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.541642 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.541912 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.541959 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:21:37.542110 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c7a4b5bacb45b98de7e31c6980e347" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926" is_pre_election: true
I20260619 10:21:37.542205 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.539423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45129 (local address 127.19.65.130:50965)
0619 10:21:37.539583 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.539591 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.539667 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.539808 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.539808 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.539810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.539876 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.539878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.540467 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.540468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.541136 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.541137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.541634 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.541635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.541767 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.541771 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.542091 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.542127 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260619 10:21:37.542251 21600 raft_consensus.cc:2454] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:37.542464 21672 leader_election.cc:302] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, 9930d2eac87b4ed197dcc42a425b7374; no voters:
I20260619 10:21:37.542446 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c7a4b5bacb45b98de7e31c6980e347" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a" is_pre_election: true
I20260619 10:21:37.542570 21880 raft_consensus.cc:2454] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:37.542595 21953 raft_consensus.cc:2781] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:37.542634 21953 raft_consensus.cc:482] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:37.542665 21953 raft_consensus.cc:3037] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.543360 21953 raft_consensus.cc:504] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.543483 21953 leader_election.cc:288] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Requested vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:37.543637 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c7a4b5bacb45b98de7e31c6980e347" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a"
I20260619 10:21:37.543727 21880 raft_consensus.cc:3037] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.544374 21880 raft_consensus.cc:2454] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:37.544502 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c7a4b5bacb45b98de7e31c6980e347" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926"
I20260619 10:21:37.544580 21672 leader_election.cc:302] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930d2eac87b4ed197dcc42a425b7374, fd60a7b7eec345b6addd9117e70c074a; no voters:
I20260619 10:21:37.544579 21600 raft_consensus.cc:3037] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.544648 21953 raft_consensus.cc:2781] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:37.544785 21953 raft_consensus.cc:686] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Becoming Leader. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Running, Role: LEADER
I20260619 10:21:37.544873 21953 consensus_queue.cc:237] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.545459 21600 raft_consensus.cc:2454] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:37.545635 21442 catalog_manager.cc:5495] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "9930d2eac87b4ed197dcc42a425b7374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } health_report { overall_health: UNKNOWN } } }
W20260619 10:21:37.561544 21787 tablet.cc:2369] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 10:21:37 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864496, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:37.586532 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.579872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39915 (local address 127.0.0.1:43476)
0619 10:21:37.580016 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.580027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.580114 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.580283 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.580284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.580606 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:37.580648 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.580650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.581354 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.581355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.581860 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.581861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.582212 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.582213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.582267 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.582795 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:37.582799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:37.584172 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:37.585687 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.585693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.585694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.585864 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.586024 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:37.586025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:37.586025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:37.586076 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:37.586267 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:37.586272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:37.586311 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.586359 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.586441 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260619 10:21:37.586540 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.579952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.19.65.190:39915)
0619 10:21:37.580003 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.580005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.580103 (+ 98us) server_negotiation.cc:407] Connection header received
0619 10:21:37.580192 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.580193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.580203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.580277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:37.580733 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.581248 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.581949 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.582131 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.582213 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.584250 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:37.584251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:37.584251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:37.584257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:37.585621 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.585922 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.585923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.585923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.585969 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:37.586125 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:37.586126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:37.586126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:37.586217 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:37.586263 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.586375 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.586480 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":26}
I20260619 10:21:37.588578 21442 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43476:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:21:37.588769 21442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:37.590145 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 442d11c6a23646d88e057e5e3250c522 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b9b7bd2156444cf9602bf1716059155]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:37.590194 21860 tablet_service.cc:1467] Processing CreateTablet for tablet 442d11c6a23646d88e057e5e3250c522 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b9b7bd2156444cf9602bf1716059155]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:37.590325 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d11c6a23646d88e057e5e3250c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.590363 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d11c6a23646d88e057e5e3250c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.590372 21720 tablet_service.cc:1467] Processing CreateTablet for tablet 442d11c6a23646d88e057e5e3250c522 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b9b7bd2156444cf9602bf1716059155]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:21:37.590533 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d11c6a23646d88e057e5e3250c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.591502 21946 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: Bootstrap starting.
I20260619 10:21:37.591888 21949 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: Bootstrap starting.
I20260619 10:21:37.591972 21947 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: Bootstrap starting.
I20260619 10:21:37.592255 21946 tablet_bootstrap.cc:654] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.592352 21947 tablet_bootstrap.cc:654] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.592432 21949 tablet_bootstrap.cc:654] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.592967 21946 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: No bootstrap required, opened a new log
I20260619 10:21:37.593019 21946 ts_tablet_manager.cc:1397] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:21:37.593191 21947 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: No bootstrap required, opened a new log
I20260619 10:21:37.593171 21946 raft_consensus.cc:348] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.593235 21946 raft_consensus.cc:374] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.593250 21947 ts_tablet_manager.cc:1397] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:37.593250 21946 raft_consensus.cc:729] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5de23b097b40b5a860631b5cfb8926, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.593310 21946 consensus_queue.cc:260] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.593405 21946 ts_tablet_manager.cc:1428] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:37.593421 21947 raft_consensus.cc:348] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.593504 21947 raft_consensus.cc:374] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.593479 21950 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.593531 21947 raft_consensus.cc:729] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.593549 21950 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.593573 21947 consensus_queue.cc:260] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.593668 21947 ts_tablet_manager.cc:1428] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.593716 21953 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.593809 21953 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.594143 21949 tablet_bootstrap.cc:492] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: No bootstrap required, opened a new log
I20260619 10:21:37.594182 21949 ts_tablet_manager.cc:1397] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:37.594345 21949 raft_consensus.cc:348] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.594436 21949 raft_consensus.cc:374] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.594475 21949 raft_consensus.cc:729] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.594527 21949 consensus_queue.cc:260] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.594609 21949 ts_tablet_manager.cc:1428] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:37.594756 21954 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.594833 21954 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.653992 21954 raft_consensus.cc:482] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:37.654095 21954 raft_consensus.cc:504] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.654383 21954 leader_election.cc:288] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017)
I20260619 10:21:37.657284 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.654614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43079 (local address 127.19.65.129:38111)
0619 10:21:37.654656 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.654658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.654811 (+ 153us) server_negotiation.cc:407] Connection header received
0619 10:21:37.654857 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.654858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.654868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.654921 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.655125 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.655669 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.656589 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.656923 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.657011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.657020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.657192 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.657225 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:37.657267 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.654550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:45017 (local address 127.19.65.131:59863)
0619 10:21:37.654602 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.654615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.654654 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.654822 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.654823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.654825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.654911 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.654913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.655676 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.655677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.656459 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.656461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.656984 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.656985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.657076 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.657079 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.657134 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.657190 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":22}
I20260619 10:21:37.657274 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.654614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59863 (local address 127.19.65.130:45017)
0619 10:21:37.654656 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.654658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.654667 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:37.654722 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.654723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.654732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.654800 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:37.654990 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.655603 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.656549 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.656909 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.656966 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.656972 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.657155 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.657200 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:37.657552 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.654550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38111 (local address 127.19.65.131:43079)
0619 10:21:37.654735 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.654746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.654802 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.654932 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.654933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.654935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:37.655048 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.655049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.655745 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.655745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.656468 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.656470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.657049 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.657050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.657098 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.657101 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.657170 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.657205 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":13,"thread_start_us":99,"threads_started":1}
I20260619 10:21:37.657616 21740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d11c6a23646d88e057e5e3250c522" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9930d2eac87b4ed197dcc42a425b7374" is_pre_election: true
I20260619 10:21:37.657738 21740 raft_consensus.cc:2454] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 0.
I20260619 10:21:37.657805 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d11c6a23646d88e057e5e3250c522" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926" is_pre_election: true
I20260619 10:21:37.657900 21600 raft_consensus.cc:2454] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 0.
I20260619 10:21:37.658054 21813 leader_election.cc:302] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930d2eac87b4ed197dcc42a425b7374, fd60a7b7eec345b6addd9117e70c074a; no voters:
I20260619 10:21:37.658167 21954 raft_consensus.cc:2781] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:37.658205 21954 raft_consensus.cc:482] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:37.658236 21954 raft_consensus.cc:3037] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.658927 21954 raft_consensus.cc:504] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.659065 21954 leader_election.cc:288] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 election: Requested vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017)
I20260619 10:21:37.659205 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d11c6a23646d88e057e5e3250c522" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926"
I20260619 10:21:37.659209 21740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d11c6a23646d88e057e5e3250c522" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9930d2eac87b4ed197dcc42a425b7374"
I20260619 10:21:37.659318 21600 raft_consensus.cc:3037] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.659338 21740 raft_consensus.cc:3037] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.659929 21600 raft_consensus.cc:2454] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 1.
I20260619 10:21:37.660075 21740 raft_consensus.cc:2454] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 1.
I20260619 10:21:37.660097 21812 leader_election.cc:302] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, fd60a7b7eec345b6addd9117e70c074a; no voters:
I20260619 10:21:37.660197 21954 raft_consensus.cc:2781] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:37.660337 21954 raft_consensus.cc:686] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [term 1 LEADER]: Becoming Leader. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Running, Role: LEADER
I20260619 10:21:37.660434 21954 consensus_queue.cc:237] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.660897 21968 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: TxnStatusTablet state changed. Reason: New leader fd60a7b7eec345b6addd9117e70c074a. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.660987 21968 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:37.661180 21970 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:21:37.661170 21442 catalog_manager.cc:5495] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd60a7b7eec345b6addd9117e70c074a (127.19.65.131). New cstate: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } health_report { overall_health: HEALTHY } } }
I20260619 10:21:37.661288 21970 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:21:37.668465 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.666195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.19.65.131:45129)
0619 10:21:37.666254 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.666256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.666264 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:37.666341 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.666341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.666357 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.666436 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.666575 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.667006 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.667731 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.667905 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.667973 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.668141 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.668248 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.668333 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.668383 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.668400 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:21:37.668505 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.666123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45129 (local address 127.0.0.1:56256)
0619 10:21:37.666200 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.666213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.666272 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.666442 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.666443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.666444 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:37.666496 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.666498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.667083 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.667084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.667647 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.667649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.667976 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.667977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.668040 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.668052 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.668316 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.668324 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.668378 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.668424 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 10:21:37.669926 21740 raft_consensus.cc:1261] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Refusing update from remote peer fd60a7b7eec345b6addd9117e70c074a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:37.669984 21600 raft_consensus.cc:1261] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Refusing update from remote peer fd60a7b7eec345b6addd9117e70c074a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 10:21:37.670219 21647 tablet.cc:2369] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:37.670222 21968 consensus_queue.cc:1035] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:37.670308 21954 consensus_queue.cc:1035] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:37.671350 21953 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: TxnStatusTablet state changed. Reason: New leader fd60a7b7eec345b6addd9117e70c074a. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.671358 21950 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: TxnStatusTablet state changed. Reason: New leader fd60a7b7eec345b6addd9117e70c074a. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.671432 21953 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.671463 21950 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.672290 21954 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.672390 21954 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:37.672469 21973 mvcc.cc:205] Tried to move back new op lower bound from 7298516982453809152 to 7298516982417690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298516982453809152}]
I20260619 10:21:37.672616 21954 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.672691 21954 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P fd60a7b7eec345b6addd9117e70c074a: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:21:37.672972 21953 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.673044 21953 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 9930d2eac87b4ed197dcc42a425b7374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.673372 21974 mvcc.cc:205] Tried to move back new op lower bound from 7298516982453809152 to 7298516982417690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298516982453809152}]
I20260619 10:21:37.673501 21950 tablet_replica.cc:439] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } } }
I20260619 10:21:37.673564 21950 tablet_replica.cc:442] T 442d11c6a23646d88e057e5e3250c522 P 3d5de23b097b40b5a860631b5cfb8926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:21:37.674693 21975 mvcc.cc:205] Tried to move back new op lower bound from 7298516982453809152 to 7298516982417690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298516982453809152}]
I20260619 10:21:37.677255 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.674454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.19.65.130:45017)
0619 10:21:37.674486 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.674487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.674494 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:21:37.674523 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.674524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.674537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.674601 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.674764 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.675480 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.676579 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.676742 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.676785 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.676957 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.677027 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.677074 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.677144 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.677179 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:21:37.681766 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.679322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45129 (local address 127.19.65.130:34059)
0619 10:21:37.679378 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.679389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.679430 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.679546 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.679546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.679547 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:37.679589 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.679590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.680387 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.680387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.680956 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.680958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.681385 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.681386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.681469 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.681482 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.681662 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.681663 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.681694 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.681722 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 10:21:37.681790 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.679382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34059 (local address 127.19.65.131:45129)
0619 10:21:37.679423 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.679425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.679433 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:37.679478 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.679479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.679488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.679541 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.679626 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.680320 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.681029 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.681314 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.681381 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.681573 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.681603 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.681655 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.681707 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.681725 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:21:37.683799 21600 raft_consensus.cc:1261] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:37.683907 21880 raft_consensus.cc:1261] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:37.683992 21953 consensus_queue.cc:1035] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:37.684103 21953 consensus_queue.cc:1035] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:37.691218 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.689080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43085 (local address 127.19.65.130:45017)
0619 10:21:37.689131 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.689133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.689178 (+ 45us) server_negotiation.cc:407] Connection header received
0619 10:21:37.689237 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.689238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.689244 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.689307 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.689418 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.689805 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.690522 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.690762 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.690819 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.690993 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.691016 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.691075 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.691145 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.691160 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:37.691246 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.689009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:45017 (local address 127.19.65.131:43085)
0619 10:21:37.689141 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.689150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.689189 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.689311 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.689311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.689312 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:37.689363 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.689364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.689886 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.689887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.690434 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.690436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.690837 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.690838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.690905 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.690918 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.691089 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.691091 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.691141 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.691178 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 10:21:37.701061 21443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:37.701148 21443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:37.701874 21443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Sending DeleteTablet for 3 replicas of tablet 23c7a4b5bacb45b98de7e31c6980e347
I20260619 10:21:37.702135 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:37 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52646
I20260619 10:21:37.702288 21860 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:37 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:56254
I20260619 10:21:37.702464 21720 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c7a4b5bacb45b98de7e31c6980e347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:37 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33576
I20260619 10:21:37.702638 21991 tablet_replica.cc:330] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: stopping tablet replica
I20260619 10:21:37.702629 21443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 10:21:37.702719 21991 raft_consensus.cc:2229] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 10:21:37.702749 21443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:37.702757 21991 raft_consensus.cc:2258] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:37.702869 21992 tablet_replica.cc:330] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: stopping tablet replica
I20260619 10:21:37.702951 21992 raft_consensus.cc:2229] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:37.703027 21992 raft_consensus.cc:2258] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:37.703673 21992 ts_tablet_manager.cc:1905] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:37.704190 21860 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (DEFAULT_TABLE table=test-table [id=c802916c20634823bebfd66e8f84e85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.704304 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.705400 21991 ts_tablet_manager.cc:1905] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:37.705608 21720 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (DEFAULT_TABLE table=test-table [id=c802916c20634823bebfd66e8f84e85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.705704 21949 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Bootstrap starting.
I20260619 10:21:37.706137 21949 tablet_bootstrap.cc:654] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.706604 21580 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (DEFAULT_TABLE table=test-table [id=c802916c20634823bebfd66e8f84e85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:37.706702 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.706825 21991 ts_tablet_manager.cc:1918] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:37.706880 21991 log.cc:1196] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/23c7a4b5bacb45b98de7e31c6980e347
I20260619 10:21:37.707109 21991 ts_tablet_manager.cc:1939] T 23c7a4b5bacb45b98de7e31c6980e347 P fd60a7b7eec345b6addd9117e70c074a: Deleting consensus metadata
I20260619 10:21:37.707487 21428 catalog_manager.cc:4841] TS fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129): tablet 23c7a4b5bacb45b98de7e31c6980e347 (table test-table [id=bce2ff7c573a4dea840530fba35d0307]) successfully deleted
I20260619 10:21:37.707931 21946 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Bootstrap starting.
I20260619 10:21:37.708302 21946 tablet_bootstrap.cc:654] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.708606 21990 tablet_replica.cc:330] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: stopping tablet replica
I20260619 10:21:37.708734 21990 raft_consensus.cc:2229] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:37.708760 21949 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: No bootstrap required, opened a new log
I20260619 10:21:37.708804 21949 ts_tablet_manager.cc:1397] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:21:37.708850 21990 raft_consensus.cc:2258] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:37.709000 21949 raft_consensus.cc:348] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.709067 21949 raft_consensus.cc:374] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.709093 21949 raft_consensus.cc:729] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.709129 21949 consensus_queue.cc:260] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.709221 21949 ts_tablet_manager.cc:1428] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.709558 21990 ts_tablet_manager.cc:1905] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:37.710984 21990 ts_tablet_manager.cc:1918] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:37.711055 21990 log.cc:1196] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/23c7a4b5bacb45b98de7e31c6980e347
I20260619 10:21:37.711325 21990 ts_tablet_manager.cc:1939] T 23c7a4b5bacb45b98de7e31c6980e347 P 3d5de23b097b40b5a860631b5cfb8926: Deleting consensus metadata
I20260619 10:21:37.711685 21946 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: No bootstrap required, opened a new log
I20260619 10:21:37.711710 21428 catalog_manager.cc:4841] TS 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111): tablet 23c7a4b5bacb45b98de7e31c6980e347 (table test-table [id=bce2ff7c573a4dea840530fba35d0307]) successfully deleted
I20260619 10:21:37.711753 21946 ts_tablet_manager.cc:1397] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 10:21:37.711771 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:37.711889 21946 raft_consensus.cc:348] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.711951 21946 raft_consensus.cc:374] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.711968 21946 raft_consensus.cc:729] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5de23b097b40b5a860631b5cfb8926, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.712005 21946 consensus_queue.cc:260] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.712073 21946 ts_tablet_manager.cc:1428] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.712514 21988 raft_consensus.cc:482] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:37.712592 21988 raft_consensus.cc:504] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.712718 21988 leader_election.cc:288] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017)
I20260619 10:21:37.713001 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd753ed06d4bdaa17f5d7f2d3ae5b2" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926" is_pre_election: true
I20260619 10:21:37.713066 21600 raft_consensus.cc:2454] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 0.
W20260619 10:21:37.713050 21813 leader_election.cc:341] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): Not found: Tablet not found: e5cd753ed06d4bdaa17f5d7f2d3ae5b2
I20260619 10:21:37.713187 21947 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Bootstrap starting.
I20260619 10:21:37.713313 21812 leader_election.cc:302] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, fd60a7b7eec345b6addd9117e70c074a; no voters: 9930d2eac87b4ed197dcc42a425b7374
I20260619 10:21:37.713325 21992 ts_tablet_manager.cc:1918] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:37.713382 21992 log.cc:1196] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/23c7a4b5bacb45b98de7e31c6980e347
I20260619 10:21:37.713518 21988 raft_consensus.cc:2781] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:37.713557 21988 raft_consensus.cc:482] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:37.713574 21947 tablet_bootstrap.cc:654] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:37.713578 21988 raft_consensus.cc:3037] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.713639 21992 ts_tablet_manager.cc:1939] T 23c7a4b5bacb45b98de7e31c6980e347 P 9930d2eac87b4ed197dcc42a425b7374: Deleting consensus metadata
I20260619 10:21:37.713994 21429 catalog_manager.cc:4841] TS 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): tablet 23c7a4b5bacb45b98de7e31c6980e347 (table test-table [id=bce2ff7c573a4dea840530fba35d0307]) successfully deleted
I20260619 10:21:37.714234 21988 raft_consensus.cc:504] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.714352 21988 leader_election.cc:288] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 election: Requested vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017)
I20260619 10:21:37.714493 21740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd753ed06d4bdaa17f5d7f2d3ae5b2" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9930d2eac87b4ed197dcc42a425b7374"
I20260619 10:21:37.714607 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd753ed06d4bdaa17f5d7f2d3ae5b2" candidate_uuid: "fd60a7b7eec345b6addd9117e70c074a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926"
I20260619 10:21:37.714666 21600 raft_consensus.cc:3037] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Advancing to term 1
W20260619 10:21:37.714690 21813 leader_election.cc:341] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): Illegal state: must be running to vote when last-logged opid is not known
I20260619 10:21:37.715255 21600 raft_consensus.cc:2454] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd60a7b7eec345b6addd9117e70c074a in term 1.
I20260619 10:21:37.715792 21812 leader_election.cc:302] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, fd60a7b7eec345b6addd9117e70c074a; no voters: 9930d2eac87b4ed197dcc42a425b7374
I20260619 10:21:37.715885 21988 raft_consensus.cc:2781] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:37.715931 21988 raft_consensus.cc:686] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 1 LEADER]: Becoming Leader. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Running, Role: LEADER
I20260619 10:21:37.715970 21988 consensus_queue.cc:237] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.716337 21443 catalog_manager.cc:5495] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd60a7b7eec345b6addd9117e70c074a (127.19.65.131). New cstate: current_term: 1 leader_uuid: "fd60a7b7eec345b6addd9117e70c074a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } health_report { overall_health: HEALTHY } } }
I20260619 10:21:37.724131 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.721206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.19.65.131:45129)
0619 10:21:37.721256 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.721258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.721265 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:21:37.721285 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.721286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.721295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.721338 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.721503 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.722219 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.723290 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.723614 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.723666 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.723812 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.723840 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.723876 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.724053 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.724068 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:21:37.724813 21947 tablet_bootstrap.cc:492] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: No bootstrap required, opened a new log
I20260619 10:21:37.724859 21947 ts_tablet_manager.cc:1397] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260619 10:21:37.725000 21947 raft_consensus.cc:348] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.725061 21947 raft_consensus.cc:374] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:37.725080 21947 raft_consensus.cc:729] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Initialized, Role: FOLLOWER
I20260619 10:21:37.725116 21947 consensus_queue.cc:260] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:37.725193 21947 ts_tablet_manager.cc:1428] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:37.728588 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.725943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45129 (local address 127.19.65.131:54751)
0619 10:21:37.725984 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:37.725994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:37.726026 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:37.726208 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:37.726208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:37.726209 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:21:37.726234 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.726235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.726982 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.726983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.727829 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:37.727831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.728257 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.728257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:37.728301 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.728304 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.728453 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.728454 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:37.728488 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 10:21:37.728521 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:21:37.728791 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:37.726083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:54751 (local address 127.19.65.131:45129)
0619 10:21:37.726108 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:37.726109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:37.726115 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:21:37.726130 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:37.726130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:37.726137 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:37.726184 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:37.726284 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.726906 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.727914 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:37.728182 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:37.728233 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:37.728362 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:37.728398 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:37.728436 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:37.728725 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 10:21:37.728739 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:21:37.730438 21740 raft_consensus.cc:3037] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:37.730520 21600 raft_consensus.cc:1261] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Refusing update from remote peer fd60a7b7eec345b6addd9117e70c074a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:37.730680 21954 consensus_queue.cc:1035] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:37.731151 21740 raft_consensus.cc:1261] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Refusing update from remote peer fd60a7b7eec345b6addd9117e70c074a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:37.731484 21954 consensus_queue.cc:1035] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:21:37.735406 21973 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 10:21:39.254707 21441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:39.254797 21441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:39.255651 21441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Sending DeleteTablet for 3 replicas of tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2
I20260619 10:21:39.255873 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52646
I20260619 10:21:39.255926 21720 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33576
I20260619 10:21:39.256064 22020 tablet_replica.cc:330] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: stopping tablet replica
I20260619 10:21:39.256108 22021 tablet_replica.cc:330] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: stopping tablet replica
I20260619 10:21:39.256117 22020 raft_consensus.cc:2229] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:39.256103 21860 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:56254
I20260619 10:21:39.256153 22020 raft_consensus.cc:2258] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.256157 22021 raft_consensus.cc:2229] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:39.256199 22021 raft_consensus.cc:2258] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.256417 22020 ts_tablet_manager.cc:1905] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.256451 22021 ts_tablet_manager.cc:1905] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.258039 22021 ts_tablet_manager.cc:1918] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:21:39.258100 22021 log.cc:1196] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/e5cd753ed06d4bdaa17f5d7f2d3ae5b2
I20260619 10:21:39.258361 22021 ts_tablet_manager.cc:1939] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 9930d2eac87b4ed197dcc42a425b7374: Deleting consensus metadata
I20260619 10:21:39.258376 22020 ts_tablet_manager.cc:1918] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:21:39.258431 22020 log.cc:1196] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/e5cd753ed06d4bdaa17f5d7f2d3ae5b2
I20260619 10:21:39.258658 22020 ts_tablet_manager.cc:1939] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P 3d5de23b097b40b5a860631b5cfb8926: Deleting consensus metadata
I20260619 10:21:39.258682 21429 catalog_manager.cc:4841] TS 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (table test-table [id=c802916c20634823bebfd66e8f84e85e]) successfully deleted
I20260619 10:21:39.258942 22022 tablet_replica.cc:330] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: stopping tablet replica
I20260619 10:21:39.258946 21428 catalog_manager.cc:4841] TS 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111): tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (table test-table [id=c802916c20634823bebfd66e8f84e85e]) successfully deleted
I20260619 10:21:39.259011 22022 raft_consensus.cc:2229] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:39.259099 22022 raft_consensus.cc:2258] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.259313 22022 ts_tablet_manager.cc:1905] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.261152 22022 ts_tablet_manager.cc:1918] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:21:39.261216 22022 log.cc:1196] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/e5cd753ed06d4bdaa17f5d7f2d3ae5b2
I20260619 10:21:39.261466 22022 ts_tablet_manager.cc:1939] T e5cd753ed06d4bdaa17f5d7f2d3ae5b2 P fd60a7b7eec345b6addd9117e70c074a: Deleting consensus metadata
I20260619 10:21:39.261801 21428 catalog_manager.cc:4841] TS fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129): tablet e5cd753ed06d4bdaa17f5d7f2d3ae5b2 (table test-table [id=c802916c20634823bebfd66e8f84e85e]) successfully deleted
Jun 19 10:21:39 dist-test-slave-xd2q krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:21:39 dist-test-slave-xd2q krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:39.280032 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:39.272769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.19.65.190:39915)
0619 10:21:39.272919 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:39.272922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:39.272980 (+ 58us) server_negotiation.cc:407] Connection header received
0619 10:21:39.273057 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:39.273058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:39.273074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:39.273144 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:39.273773 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:39.274269 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:39.275202 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:39.275465 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:39.275519 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:39.277768 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:39.277770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:39.277770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:39.277775 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:39.279141 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:39.279482 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:39.279483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:39.279483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:39.279530 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:39.279663 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:39.279663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:39.279663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:39.279743 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:39.279806 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:39.279896 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 10:21:39.279960 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260619 10:21:39.281528 21441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:39.281687 21441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:39.282888 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 269705e07f974de1b1d73174cede6adf (DEFAULT_TABLE table=test-table [id=28fb52af06f647978ea8028ec5263ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.282927 21860 tablet_service.cc:1467] Processing CreateTablet for tablet 269705e07f974de1b1d73174cede6adf (DEFAULT_TABLE table=test-table [id=28fb52af06f647978ea8028ec5263ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.283033 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269705e07f974de1b1d73174cede6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.283042 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269705e07f974de1b1d73174cede6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.283396 21720 tablet_service.cc:1467] Processing CreateTablet for tablet 269705e07f974de1b1d73174cede6adf (DEFAULT_TABLE table=test-table [id=28fb52af06f647978ea8028ec5263ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.283510 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269705e07f974de1b1d73174cede6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.284184 22031 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Bootstrap starting.
I20260619 10:21:39.284425 22032 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Bootstrap starting.
I20260619 10:21:39.284603 22031 tablet_bootstrap.cc:654] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.284844 22032 tablet_bootstrap.cc:654] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.284888 22033 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Bootstrap starting.
I20260619 10:21:39.285321 22033 tablet_bootstrap.cc:654] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.285418 22031 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: No bootstrap required, opened a new log
I20260619 10:21:39.285471 22031 ts_tablet_manager.cc:1397] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:39.285626 22031 raft_consensus.cc:348] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.285713 22031 raft_consensus.cc:374] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.285740 22031 raft_consensus.cc:729] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5de23b097b40b5a860631b5cfb8926, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.285821 22031 consensus_queue.cc:260] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.285919 22032 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: No bootstrap required, opened a new log
I20260619 10:21:39.285949 22033 tablet_bootstrap.cc:492] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: No bootstrap required, opened a new log
I20260619 10:21:39.285972 22032 ts_tablet_manager.cc:1397] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:39.285996 22033 ts_tablet_manager.cc:1397] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:39.285997 22031 ts_tablet_manager.cc:1428] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:39.286072 22032 raft_consensus.cc:348] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.286134 22032 raft_consensus.cc:374] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.286103 22033 raft_consensus.cc:348] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.286152 22032 raft_consensus.cc:729] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.286167 22033 raft_consensus.cc:374] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.286192 22033 raft_consensus.cc:729] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.286216 22032 consensus_queue.cc:260] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.286239 22033 consensus_queue.cc:260] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.286316 22032 ts_tablet_manager.cc:1428] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:39.286401 22033 ts_tablet_manager.cc:1428] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:39.382629 22038 raft_consensus.cc:482] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:39.382712 22038 raft_consensus.cc:504] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.382853 22038 leader_election.cc:288] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:39.383040 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269705e07f974de1b1d73174cede6adf" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926" is_pre_election: true
I20260619 10:21:39.383138 21600 raft_consensus.cc:2454] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:39.383145 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269705e07f974de1b1d73174cede6adf" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a" is_pre_election: true
I20260619 10:21:39.383270 21880 raft_consensus.cc:2454] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:39.383340 21672 leader_election.cc:302] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, 9930d2eac87b4ed197dcc42a425b7374; no voters:
I20260619 10:21:39.383451 22038 raft_consensus.cc:2781] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:39.383491 22038 raft_consensus.cc:482] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:39.383512 22038 raft_consensus.cc:3037] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.384171 22038 raft_consensus.cc:504] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.384300 22038 leader_election.cc:288] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Requested vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:39.384456 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269705e07f974de1b1d73174cede6adf" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926"
I20260619 10:21:39.384464 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269705e07f974de1b1d73174cede6adf" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a"
I20260619 10:21:39.384536 21880 raft_consensus.cc:3037] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.384548 21600 raft_consensus.cc:3037] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.385157 21880 raft_consensus.cc:2454] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:39.385248 21600 raft_consensus.cc:2454] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:39.385346 21672 leader_election.cc:302] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930d2eac87b4ed197dcc42a425b7374, fd60a7b7eec345b6addd9117e70c074a; no voters:
I20260619 10:21:39.385458 22038 raft_consensus.cc:2781] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:39.385602 22038 raft_consensus.cc:686] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Becoming Leader. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Running, Role: LEADER
I20260619 10:21:39.385644 22038 consensus_queue.cc:237] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.386229 21441 catalog_manager.cc:5495] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "9930d2eac87b4ed197dcc42a425b7374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20260619 10:21:39.403168 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:39.400871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.19.65.130:45017)
0619 10:21:39.401021 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:39.401023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:39.401029 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:21:39.401055 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:39.401056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:39.401066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:39.401138 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:39.401283 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:39.401768 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:39.402495 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:39.402706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:39.402787 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:39.402932 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:39.402963 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:39.403001 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:39.403050 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:21:39.403070 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260619 10:21:39.405284 21880 raft_consensus.cc:1261] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:39.405323 21600 raft_consensus.cc:1261] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:39.405501 22039 consensus_queue.cc:1035] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:39.405594 22038 consensus_queue.cc:1035] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:39.414988 21441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:39.415071 21441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:39.415719 21441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Sending DeleteTablet for 3 replicas of tablet 269705e07f974de1b1d73174cede6adf
I20260619 10:21:39.415959 21720 tablet_service.cc:1514] Processing DeleteTablet for tablet 269705e07f974de1b1d73174cede6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33576
I20260619 10:21:39.415949 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 269705e07f974de1b1d73174cede6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52646
I20260619 10:21:39.416045 22021 tablet_replica.cc:330] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: stopping tablet replica
I20260619 10:21:39.416062 21860 tablet_service.cc:1514] Processing DeleteTablet for tablet 269705e07f974de1b1d73174cede6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:39 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:56254
I20260619 10:21:39.416105 22021 raft_consensus.cc:2229] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:39.416147 22022 tablet_replica.cc:330] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: stopping tablet replica
I20260619 10:21:39.416158 22021 raft_consensus.cc:2258] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.416193 22022 raft_consensus.cc:2229] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:39.416163 21441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 10:21:39.416234 22020 tablet_replica.cc:330] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: stopping tablet replica
I20260619 10:21:39.416249 22022 raft_consensus.cc:2258] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.416277 22020 raft_consensus.cc:2229] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 10:21:39.416285 21441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:39.416325 22020 raft_consensus.cc:2258] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:39.416678 22020 ts_tablet_manager.cc:1905] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.416678 22022 ts_tablet_manager.cc:1905] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.416733 22021 ts_tablet_manager.cc:1905] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:39.418051 22021 ts_tablet_manager.cc:1918] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:39.418097 22021 log.cc:1196] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/269705e07f974de1b1d73174cede6adf
I20260619 10:21:39.418073 22022 ts_tablet_manager.cc:1918] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:39.418073 22020 ts_tablet_manager.cc:1918] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:21:39.418131 22022 log.cc:1196] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/269705e07f974de1b1d73174cede6adf
I20260619 10:21:39.418131 22020 log.cc:1196] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/269705e07f974de1b1d73174cede6adf
I20260619 10:21:39.418277 22021 ts_tablet_manager.cc:1939] T 269705e07f974de1b1d73174cede6adf P 9930d2eac87b4ed197dcc42a425b7374: Deleting consensus metadata
I20260619 10:21:39.418341 22020 ts_tablet_manager.cc:1939] T 269705e07f974de1b1d73174cede6adf P 3d5de23b097b40b5a860631b5cfb8926: Deleting consensus metadata
I20260619 10:21:39.418341 22022 ts_tablet_manager.cc:1939] T 269705e07f974de1b1d73174cede6adf P fd60a7b7eec345b6addd9117e70c074a: Deleting consensus metadata
I20260619 10:21:39.418581 21429 catalog_manager.cc:4841] TS 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): tablet 269705e07f974de1b1d73174cede6adf (table test-table [id=28fb52af06f647978ea8028ec5263ac3]) successfully deleted
I20260619 10:21:39.418838 21428 catalog_manager.cc:4841] TS 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111): tablet 269705e07f974de1b1d73174cede6adf (table test-table [id=28fb52af06f647978ea8028ec5263ac3]) successfully deleted
I20260619 10:21:39.418905 21428 catalog_manager.cc:4841] TS fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129): tablet 269705e07f974de1b1d73174cede6adf (table test-table [id=28fb52af06f647978ea8028ec5263ac3]) successfully deleted
I20260619 10:21:39.419275 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 009bfed5742146da80904100d2f21bd8 (DEFAULT_TABLE table=test-table [id=8c47140f49d24e6984e1c3cbe07c096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.419324 21720 tablet_service.cc:1467] Processing CreateTablet for tablet 009bfed5742146da80904100d2f21bd8 (DEFAULT_TABLE table=test-table [id=8c47140f49d24e6984e1c3cbe07c096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.419327 21860 tablet_service.cc:1467] Processing CreateTablet for tablet 009bfed5742146da80904100d2f21bd8 (DEFAULT_TABLE table=test-table [id=8c47140f49d24e6984e1c3cbe07c096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:39.419389 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009bfed5742146da80904100d2f21bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.419409 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009bfed5742146da80904100d2f21bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.419417 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009bfed5742146da80904100d2f21bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:39.420171 22032 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Bootstrap starting.
I20260619 10:21:39.420419 22031 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926: Bootstrap starting.
I20260619 10:21:39.420440 22033 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Bootstrap starting.
I20260619 10:21:39.420475 22032 tablet_bootstrap.cc:654] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.420723 22033 tablet_bootstrap.cc:654] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.420995 22032 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: No bootstrap required, opened a new log
I20260619 10:21:39.420996 22031 tablet_bootstrap.cc:654] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:39.421036 22032 ts_tablet_manager.cc:1397] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:39.421129 22032 raft_consensus.cc:348] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.421169 22032 raft_consensus.cc:374] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.421181 22032 raft_consensus.cc:729] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd60a7b7eec345b6addd9117e70c074a, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.421242 22032 consensus_queue.cc:260] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.421346 22032 ts_tablet_manager.cc:1428] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:39.421526 22031 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926: No bootstrap required, opened a new log
I20260619 10:21:39.421600 22031 ts_tablet_manager.cc:1397] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:39.421738 22031 raft_consensus.cc:348] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.421813 22033 tablet_bootstrap.cc:492] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: No bootstrap required, opened a new log
I20260619 10:21:39.421824 22031 raft_consensus.cc:374] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.421856 22031 raft_consensus.cc:729] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5de23b097b40b5a860631b5cfb8926, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.421855 22033 ts_tablet_manager.cc:1397] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:39.421912 22031 consensus_queue.cc:260] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.421999 22031 ts_tablet_manager.cc:1428] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:39.421980 22033 raft_consensus.cc:348] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.422042 22033 raft_consensus.cc:374] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:39.422063 22033 raft_consensus.cc:729] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Initialized, Role: FOLLOWER
I20260619 10:21:39.422130 22033 consensus_queue.cc:260] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.422230 22033 ts_tablet_manager.cc:1428] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 10:21:39.452386 21927 tablet.cc:2369] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:39.474192 22039 raft_consensus.cc:482] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:39.474273 22039 raft_consensus.cc:504] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.474449 22039 leader_election.cc:288] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:39.474658 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009bfed5742146da80904100d2f21bd8" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926" is_pre_election: true
I20260619 10:21:39.474654 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009bfed5742146da80904100d2f21bd8" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a" is_pre_election: true
I20260619 10:21:39.474732 21600 raft_consensus.cc:2454] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:39.474772 21880 raft_consensus.cc:2454] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 0.
I20260619 10:21:39.474891 21672 leader_election.cc:302] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5de23b097b40b5a860631b5cfb8926, 9930d2eac87b4ed197dcc42a425b7374; no voters:
I20260619 10:21:39.475008 22039 raft_consensus.cc:2781] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:39.475103 22039 raft_consensus.cc:482] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:39.475139 22039 raft_consensus.cc:3037] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.476058 22039 raft_consensus.cc:504] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.476202 22039 leader_election.cc:288] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Requested vote from peers 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111), fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129)
I20260619 10:21:39.476378 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009bfed5742146da80904100d2f21bd8" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5de23b097b40b5a860631b5cfb8926"
I20260619 10:21:39.476370 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009bfed5742146da80904100d2f21bd8" candidate_uuid: "9930d2eac87b4ed197dcc42a425b7374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd60a7b7eec345b6addd9117e70c074a"
I20260619 10:21:39.476450 21880 raft_consensus.cc:3037] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.476472 21600 raft_consensus.cc:3037] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:39.477005 21600 raft_consensus.cc:2454] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:39.477077 21880 raft_consensus.cc:2454] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930d2eac87b4ed197dcc42a425b7374 in term 1.
I20260619 10:21:39.477274 21672 leader_election.cc:302] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930d2eac87b4ed197dcc42a425b7374, fd60a7b7eec345b6addd9117e70c074a; no voters:
I20260619 10:21:39.477404 22039 raft_consensus.cc:2781] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:39.477478 22039 raft_consensus.cc:686] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Becoming Leader. State: Replica: 9930d2eac87b4ed197dcc42a425b7374, State: Running, Role: LEADER
I20260619 10:21:39.477526 22039 consensus_queue.cc:237] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } }
I20260619 10:21:39.478080 21442 catalog_manager.cc:5495] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "9930d2eac87b4ed197dcc42a425b7374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930d2eac87b4ed197dcc42a425b7374" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20260619 10:21:39.497419 21880 raft_consensus.cc:1261] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:39.497467 21600 raft_consensus.cc:1261] T 009bfed5742146da80904100d2f21bd8 P 3d5de23b097b40b5a860631b5cfb8926 [term 1 FOLLOWER]: Refusing update from remote peer 9930d2eac87b4ed197dcc42a425b7374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:39.497611 22039 consensus_queue.cc:1035] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd60a7b7eec345b6addd9117e70c074a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:39.497684 22038 consensus_queue.cc:1035] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5de23b097b40b5a860631b5cfb8926" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:21:39.501734 22003 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 10:21:41.017598 21443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:41.017684 21443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:41.018496 21443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a51da65e82aa410f9f849e8505409682: Sending DeleteTablet for 3 replicas of tablet 009bfed5742146da80904100d2f21bd8
I20260619 10:21:41.018740 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 009bfed5742146da80904100d2f21bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:41 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52646
I20260619 10:21:41.018757 21720 tablet_service.cc:1514] Processing DeleteTablet for tablet 009bfed5742146da80904100d2f21bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:41 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33576
I20260619 10:21:41.018931 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21515
I20260619 10:21:41.018958 22075 tablet_replica.cc:330] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: stopping tablet replica
I20260619 10:21:41.018956 21860 tablet_service.cc:1514] Processing DeleteTablet for tablet 009bfed5742146da80904100d2f21bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:41 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:56254
I20260619 10:21:41.019598 22076 tablet_replica.cc:330] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: stopping tablet replica
I20260619 10:21:41.019680 22076 raft_consensus.cc:2229] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:41.019731 22076 raft_consensus.cc:2258] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:41.019990 22076 ts_tablet_manager.cc:1905] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:41.020108 22075 raft_consensus.cc:2229] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:41.020211 22075 raft_consensus.cc:2258] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:41.020459 22075 ts_tablet_manager.cc:1905] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:41.021515 22076 ts_tablet_manager.cc:1918] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:21:41.021580 22076 log.cc:1196] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/009bfed5742146da80904100d2f21bd8
I20260619 10:21:41.021808 22076 ts_tablet_manager.cc:1939] T 009bfed5742146da80904100d2f21bd8 P fd60a7b7eec345b6addd9117e70c074a: Deleting consensus metadata
I20260619 10:21:41.021986 22075 ts_tablet_manager.cc:1918] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:21:41.022060 22075 log.cc:1196] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/009bfed5742146da80904100d2f21bd8
I20260619 10:21:41.022152 21428 catalog_manager.cc:4841] TS fd60a7b7eec345b6addd9117e70c074a (127.19.65.131:45129): tablet 009bfed5742146da80904100d2f21bd8 (table test-table [id=8c47140f49d24e6984e1c3cbe07c096f]) successfully deleted
I20260619 10:21:41.022320 22075 ts_tablet_manager.cc:1939] T 009bfed5742146da80904100d2f21bd8 P 9930d2eac87b4ed197dcc42a425b7374: Deleting consensus metadata
I20260619 10:21:41.022656 21429 catalog_manager.cc:4841] TS 9930d2eac87b4ed197dcc42a425b7374 (127.19.65.130:45017): tablet 009bfed5742146da80904100d2f21bd8 (table test-table [id=8c47140f49d24e6984e1c3cbe07c096f]) successfully deleted
W20260619 10:21:41.024518 21428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:38111 (error 108)
I20260619 10:21:41.025072 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21655
I20260619 10:21:41.025108 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:41.024893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38111 (local address 127.0.0.1:51394)
0619 10:21:41.025016 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:41.025065 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
W20260619 10:21:41.025187 21428 catalog_manager.cc:4584] TS 3d5de23b097b40b5a860631b5cfb8926 (127.19.65.129:38111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 009bfed5742146da80904100d2f21bd8: Network error: Client connection negotiation failed: client connection to 127.19.65.129:38111: connect: Connection refused (error 111)
I20260619 10:21:41.030844 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21795
I20260619 10:21:41.036326 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 21411
2026-06-19T10:21:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5326 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:41 dist-test-slave-xd2q krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:41 dist-test-slave-xd2q krb5kdc[22081](info): set up 2 sockets
Jun 19 10:21:41 dist-test-slave-xd2q krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260619 10:21:42.062923 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:42.171839 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45147
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:42.250751 22094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:42.250984 22094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:42.251006 22094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:42.252163 22094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:42.252202 22094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:42.252218 22094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:42.252270 22094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:42.252288 22094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:42.253697 22094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:42.253934 22094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:42.254194 22094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:42.258927 22100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.258975 22102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.259006 22099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:42.259327 22094 server_base.cc:1034] running on GCE node
I20260619 10:21:42.259529 22094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:42.259749 22094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:42.260893 22094 hybrid_clock.cc:648] HybridClock initialized: now 1781864502260876 us; error 33 us; skew 500 ppm
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:42.263211 22094 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:42.263633 22094 webserver.cc:460] Webserver started at http://127.19.65.190:35293/ using document root <none> and password file <none>
I20260619 10:21:42.263857 22094 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:42.263932 22094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:42.264078 22094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:42.265048 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "0c3248b02cf34b29b64749ad064fc195"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "272c990ba1a7d4d0bd48960272de6c9d"
server_key_iv: "7f7ce259810a5b526300dee5c997ad09"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.265420 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "0c3248b02cf34b29b64749ad064fc195"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "272c990ba1a7d4d0bd48960272de6c9d"
server_key_iv: "7f7ce259810a5b526300dee5c997ad09"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.266714 22094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:42.267449 22109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.267616 22094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:42.267711 22094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "0c3248b02cf34b29b64749ad064fc195"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "272c990ba1a7d4d0bd48960272de6c9d"
server_key_iv: "7f7ce259810a5b526300dee5c997ad09"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.267793 22094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:42.290354 22094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:42.292330 22094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:42.292490 22094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:42.296276 22094 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45147
I20260619 10:21:42.296316 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:45147 every 8 connection(s)
I20260619 10:21:42.296693 22094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:42.297530 22162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:42.299746 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195: Bootstrap starting.
I20260619 10:21:42.300303 22162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:42.300621 22162 log.cc:824] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:42.301292 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195: No bootstrap required, opened a new log
I20260619 10:21:42.302806 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } }
I20260619 10:21:42.302939 22162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:42.302961 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3248b02cf34b29b64749ad064fc195, State: Initialized, Role: FOLLOWER
I20260619 10:21:42.303150 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } }
I20260619 10:21:42.303231 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:42.303274 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:42.303349 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:42.303872 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } }
I20260619 10:21:42.303995 22162 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3248b02cf34b29b64749ad064fc195; no voters:
I20260619 10:21:42.304172 22162 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:42.304193 22165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:42.304365 22165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3248b02cf34b29b64749ad064fc195, State: Running, Role: LEADER
I20260619 10:21:42.304409 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:42.304483 22165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } }
I20260619 10:21:42.304844 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3248b02cf34b29b64749ad064fc195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } } }
I20260619 10:21:42.304847 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3248b02cf34b29b64749ad064fc195. Latest consensus state: current_term: 1 leader_uuid: "0c3248b02cf34b29b64749ad064fc195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3248b02cf34b29b64749ad064fc195" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45147 } } }
I20260619 10:21:42.304981 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:42.305042 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:42.305246 22170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:42.305840 22170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:42.306293 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22094
I20260619 10:21:42.306385 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:42.306577 19718 external_mini_cluster.cc:1428] Setting key 0d06b3218b8dfefa9762bc2858f446b7
I20260619 10:21:42.307549 22170 catalog_manager.cc:1347] Generated new cluster ID: 2393760123844050bb520dc3b39f77a0
I20260619 10:21:42.307600 22170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:42.317968 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.306985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.19.65.190:45147)
0619 10:21:42.307667 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:42.307676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:42.307707 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:21:42.307823 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:42.307830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:42.307896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:42.308033 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:42.308720 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.310109 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.311105 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.311720 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.311828 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.314464 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:42.314468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:42.314468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:42.314478 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:42.316372 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.316760 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.316762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.316763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.316853 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.317048 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.317049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.317049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.317303 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:42.317436 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:42.317553 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 10:21:42.317758 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":83,"threads_started":1}
I20260619 10:21:42.332165 22170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:42.332811 22170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:42.337863 22170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3248b02cf34b29b64749ad064fc195: Generated new TSK 0
I20260619 10:21:42.338029 22170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:42.366026 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:42.442232 22188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:42.442420 22188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:42.442441 22188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:42.443904 22188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:42.443949 22188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:42.443976 22188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:42.445506 22188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:42.445740 22188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:42.445973 22188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:42.450778 22194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.450790 22196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:42.451032 22188 server_base.cc:1034] running on GCE node
W20260619 10:21:42.450790 22193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:42.451349 22188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:42.451619 22188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:42.452826 22188 hybrid_clock.cc:648] HybridClock initialized: now 1781864502452798 us; error 28 us; skew 500 ppm
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:42.455474 22188 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:42.455916 22188 webserver.cc:460] Webserver started at http://127.19.65.129:32849/ using document root <none> and password file <none>
I20260619 10:21:42.456156 22188 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:42.456226 22188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:42.456360 22188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:42.457386 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "f3520ed81d394d3087cac4081771794b"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "08a550e8bca2560a19c90beed3dc1500"
server_key_iv: "9c4edde69c23953f570ce28c24fa95b6"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.457757 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "f3520ed81d394d3087cac4081771794b"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "08a550e8bca2560a19c90beed3dc1500"
server_key_iv: "9c4edde69c23953f570ce28c24fa95b6"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.459043 22188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:21:42.459882 22203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.460072 22188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.460178 22188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "f3520ed81d394d3087cac4081771794b"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "08a550e8bca2560a19c90beed3dc1500"
server_key_iv: "9c4edde69c23953f570ce28c24fa95b6"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.460271 22188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:42.470095 22188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:42.472268 22188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:42.472420 22188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:42.472618 22188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:42.472906 22188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:42.472956 22188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.472994 22188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:42.473019 22188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.478322 22188 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:41895
I20260619 10:21:42.478348 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:41895 every 8 connection(s)
I20260619 10:21:42.478662 22188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:42.480288 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22188
I20260619 10:21:42.480407 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:42.480587 19718 external_mini_cluster.cc:1428] Setting key 228f7ac296887c2033e321c4f9f63f2a
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:42.488839 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.479391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:38303 (local address 127.19.65.190:45147)
0619 10:21:42.479456 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:42.479458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:42.479684 (+ 226us) server_negotiation.cc:407] Connection header received
0619 10:21:42.480199 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:42.480200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:42.480213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:42.480268 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:42.481311 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.481824 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.483252 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.483562 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.483622 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.486054 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:42.486055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:42.486055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:42.486061 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:42.487442 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.487917 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.487918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.487918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.487965 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.488228 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.488229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.488229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.488318 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:42.488362 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:42.488713 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 10:21:42.488780 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:21:42.489105 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.479399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45147 (local address 127.19.65.129:38303)
0619 10:21:42.479557 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:42.479588 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:42.480068 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:42.480329 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:42.480333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:42.480999 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:42.481228 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.481235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.481914 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.481915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.483161 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.483164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.483636 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.483637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.483761 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.484235 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:42.484242 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:42.485963 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:42.487518 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.487525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.487527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.487845 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.488027 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.488028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.488028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.488158 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.488378 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:42.488384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:42.488561 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 10:21:42.488696 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 10:21:42.488875 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:21:42.489643 22317 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45147
I20260619 10:21:42.489732 22317 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:42.489919 22317 heartbeater.cc:511] Master 127.19.65.190:45147 requested a full tablet report, sending...
I20260619 10:21:42.490438 22126 ts_manager.cc:194] Registered new tserver with Master: f3520ed81d394d3087cac4081771794b (127.19.65.129:41895)
I20260619 10:21:42.491029 22126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:38303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:42.505481 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:42.581878 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:42.582093 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:42.582129 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:42.583281 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:42.583374 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:42.583433 22322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:42.584808 22322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:42.585054 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:42.585265 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:42.589989 22327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.589982 22328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.590009 22330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:42.590101 22322 server_base.cc:1034] running on GCE node
I20260619 10:21:42.590436 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:42.590695 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:42.591857 22322 hybrid_clock.cc:648] HybridClock initialized: now 1781864502591843 us; error 24 us; skew 500 ppm
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:42.594451 22322 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:42.594878 22322 webserver.cc:460] Webserver started at http://127.19.65.130:36779/ using document root <none> and password file <none>
I20260619 10:21:42.595081 22322 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:42.595120 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:42.595203 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:42.596206 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "f3edfd9339084f41ad2a7c3608015a82"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "a501fe0f47a8da222538ee9bca88d31d"
server_key_iv: "7794842886f660aa6acecbd75f98a7da"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.596495 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "f3edfd9339084f41ad2a7c3608015a82"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "a501fe0f47a8da222538ee9bca88d31d"
server_key_iv: "7794842886f660aa6acecbd75f98a7da"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.597677 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:42.598276 22337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.598435 22322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:42.598516 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "f3edfd9339084f41ad2a7c3608015a82"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "a501fe0f47a8da222538ee9bca88d31d"
server_key_iv: "7794842886f660aa6acecbd75f98a7da"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.598565 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:42.618746 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:42.620528 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:42.620625 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:42.620775 22322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:42.621011 22322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:42.621039 22322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.621059 22322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:42.621071 22322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.626331 22322 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34095
I20260619 10:21:42.626413 22450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:34095 every 8 connection(s)
I20260619 10:21:42.626714 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:42.629849 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22322
I20260619 10:21:42.629938 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:42.630125 19718 external_mini_cluster.cc:1428] Setting key 8f2bd4256d82f0080f12c4b1e0a2f937
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:42.635501 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.627413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:45979 (local address 127.19.65.190:45147)
0619 10:21:42.627464 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:42.627466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:42.627648 (+ 182us) server_negotiation.cc:407] Connection header received
0619 10:21:42.628084 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:42.628085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:42.628093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:42.628143 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:42.628936 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.629389 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.630494 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.630759 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.630839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.632834 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:42.632834 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:42.632835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:42.632841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:42.634126 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.634506 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.634507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.634507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.634587 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.634835 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.634836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.634836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.634952 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:42.635034 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:42.635323 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 10:21:42.635404 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:21:42.635666 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.627390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45147 (local address 127.19.65.130:45979)
0619 10:21:42.627533 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:42.627570 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:42.628011 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:42.628188 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:42.628191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:42.628678 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:42.628866 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.628870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.629471 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.629471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.630444 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.630446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.630872 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.630872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.630964 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.631486 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:42.631494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:42.632759 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:42.634212 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.634218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.634219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.634424 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.634654 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.634655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.634656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.634764 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.635025 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:42.635031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:42.635177 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 10:21:42.635281 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 10:21:42.635450 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260619 10:21:42.636139 22451 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45147
I20260619 10:21:42.636236 22451 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:42.636417 22451 heartbeater.cc:511] Master 127.19.65.190:45147 requested a full tablet report, sending...
I20260619 10:21:42.636785 22126 ts_manager.cc:194] Registered new tserver with Master: f3edfd9339084f41ad2a7c3608015a82 (127.19.65.130:34095)
I20260619 10:21:42.637288 22126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:45979
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:42.654350 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:42.731812 22456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:42.731984 22456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:42.732004 22456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:42.733126 22456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:42.733165 22456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:42.733191 22456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:42.734576 22456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:42.734830 22456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:42.735065 22456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:42.739916 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.739988 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:42.739936 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:42.740095 22456 server_base.cc:1034] running on GCE node
I20260619 10:21:42.740329 22456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:42.740516 22456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:42.741671 22456 hybrid_clock.cc:648] HybridClock initialized: now 1781864502741656 us; error 28 us; skew 500 ppm
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:42.744086 22456 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:42.744534 22456 webserver.cc:460] Webserver started at http://127.19.65.131:40053/ using document root <none> and password file <none>
I20260619 10:21:42.744763 22456 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:42.744835 22456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:42.744992 22456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:42.745990 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "9fd3dd7485774251b25d651c3d3ff478"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "1ceaa5be23f3ee235a7e3b0096db3799"
server_key_iv: "2b3671f1cdbb0a9d4bfd4e18cbc616bd"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.746337 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "9fd3dd7485774251b25d651c3d3ff478"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "1ceaa5be23f3ee235a7e3b0096db3799"
server_key_iv: "2b3671f1cdbb0a9d4bfd4e18cbc616bd"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.747690 22456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:42.748426 22471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.748617 22456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:42.748723 22456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "9fd3dd7485774251b25d651c3d3ff478"
format_stamp: "Formatted at 2026-06-19 10:21:42 on dist-test-slave-xd2q"
server_key: "1ceaa5be23f3ee235a7e3b0096db3799"
server_key_iv: "2b3671f1cdbb0a9d4bfd4e18cbc616bd"
server_key_version: "encryptionkey@0"
I20260619 10:21:42.748824 22456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:42.794013 22456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:42.796440 22456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:42.796612 22456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:42.796875 22456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:42.797189 22456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:42.797242 22456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.797276 22456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:42.797302 22456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:42.803351 22456 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45711
I20260619 10:21:42.803385 22584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45711 every 8 connection(s)
I20260619 10:21:42.803700 22456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:42.808890 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22456
I20260619 10:21:42.809002 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:42.809237 19718 external_mini_cluster.cc:1428] Setting key 36c08f9409d9c4097054112abcf11db3
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864502, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:42.813726 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.804407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43739 (local address 127.19.65.190:45147)
0619 10:21:42.804446 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:42.804448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:42.804717 (+ 269us) server_negotiation.cc:407] Connection header received
0619 10:21:42.805295 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:42.805296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:42.805318 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:42.805425 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:42.806544 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.807048 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.808223 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.808413 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.808478 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.811063 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:42.811064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:42.811064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:42.811072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:42.812649 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.813003 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.813004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.813004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.813052 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:42.813251 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:42.813251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:42.813252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:42.813348 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:42.813397 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:42.813610 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 10:21:42.813666 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:21:42.813977 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:42.804441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45147 (local address 127.19.65.131:43739)
0619 10:21:42.804582 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:42.804636 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:42.805180 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:42.805494 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:42.805498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:42.806208 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:42.806461 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.806466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.807168 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.807169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.808122 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:42.808125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:42.808494 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:42.808495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:42.808593 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:42.809178 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:42.809189 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:42.810966 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:42.812720 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.812726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.812727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.812938 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.813107 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:42.813107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:42.813107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:42.813199 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:42.813403 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:42.813406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:42.813513 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 10:21:42.813596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:21:42.813705 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260619 10:21:42.814476 22585 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45147
I20260619 10:21:42.814590 22585 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:42.814764 22585 heartbeater.cc:511] Master 127.19.65.190:45147 requested a full tablet report, sending...
I20260619 10:21:42.815147 22126 ts_manager.cc:194] Registered new tserver with Master: 9fd3dd7485774251b25d651c3d3ff478 (127.19.65.131:45711)
I20260619 10:21:42.815539 22126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:43739
I20260619 10:21:42.820945 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:21:42.831236 22184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:42.830460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.19.65.190:45147)
0619 10:21:42.830517 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:42.830519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:42.830592 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:21:42.830633 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:42.830634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:42.830642 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:42.830688 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:42.831165 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48500: BlockingRecv error: recv got EOF from 127.0.0.1:48500 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:21:42.832306 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22188
I20260619 10:21:42.839913 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22322
I20260619 10:21:42.844365 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22456
I20260619 10:21:42.848366 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22094
2026-06-19T10:21:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1811 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22597](info): set up 2 sockets
Jun 19 10:21:42 dist-test-slave-xd2q krb5kdc[22597](info): commencing operation
krb5kdc: starting...
W20260619 10:21:43.871075 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:43 dist-test-slave-xd2q krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:43.978193 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45837
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:44.057986 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:44.058166 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:44.058187 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:44.059330 22610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:44.059367 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:44.059381 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:44.059399 22610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:44.059450 22610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:44.060824 22610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:44.061048 22610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:44.061256 22610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:44.065826 22615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:44.065815 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:44.065841 22610 server_base.cc:1034] running on GCE node
W20260619 10:21:44.065815 22616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:44.066252 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:44.066524 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:44.067653 22610 hybrid_clock.cc:648] HybridClock initialized: now 1781864504067627 us; error 43 us; skew 500 ppm
Jun 19 10:21:44 dist-test-slave-xd2q krb5kdc[22597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864504, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:44.069865 22610 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:44.070284 22610 webserver.cc:460] Webserver started at http://127.19.65.190:43165/ using document root <none> and password file <none>
I20260619 10:21:44.070490 22610 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:44.070546 22610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:44.070679 22610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:44.071620 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "60dcf120fb9142ed8b25d942cb07098f"
format_stamp: "Formatted at 2026-06-19 10:21:44 on dist-test-slave-xd2q"
server_key: "8e81c715a066ce51abd648c613347172"
server_key_iv: "2d47a1e203d495d9a9b2ddabb758a274"
server_key_version: "encryptionkey@0"
I20260619 10:21:44.071942 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "60dcf120fb9142ed8b25d942cb07098f"
format_stamp: "Formatted at 2026-06-19 10:21:44 on dist-test-slave-xd2q"
server_key: "8e81c715a066ce51abd648c613347172"
server_key_iv: "2d47a1e203d495d9a9b2ddabb758a274"
server_key_version: "encryptionkey@0"
I20260619 10:21:44.073130 22610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:44.073771 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:44.073932 22610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:44.074024 22610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "60dcf120fb9142ed8b25d942cb07098f"
format_stamp: "Formatted at 2026-06-19 10:21:44 on dist-test-slave-xd2q"
server_key: "8e81c715a066ce51abd648c613347172"
server_key_iv: "2d47a1e203d495d9a9b2ddabb758a274"
server_key_version: "encryptionkey@0"
I20260619 10:21:44.074103 22610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:44.084195 22610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:44.085745 22610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:44.085891 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:44.089872 22610 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45837
I20260619 10:21:44.089900 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:45837 every 8 connection(s)
I20260619 10:21:44.090365 22610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:44.090951 22678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:44.091904 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22610
I20260619 10:21:44.092010 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:44.092180 19718 external_mini_cluster.cc:1428] Setting key a4abed3f8a4ce47b81fc62ec391e5b58
I20260619 10:21:44.094172 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f: Bootstrap starting.
I20260619 10:21:44.094723 22678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:44.094987 22678 log.cc:824] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:44.095781 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f: No bootstrap required, opened a new log
I20260619 10:21:44.098275 22678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } }
I20260619 10:21:44.098395 22678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:44.098417 22678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dcf120fb9142ed8b25d942cb07098f, State: Initialized, Role: FOLLOWER
I20260619 10:21:44.098536 22678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } }
I20260619 10:21:44.098608 22678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:44.098630 22678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:44.098690 22678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:21:44 dist-test-slave-xd2q krb5kdc[22597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:44.099336 22678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } }
I20260619 10:21:44.099472 22678 leader_election.cc:302] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60dcf120fb9142ed8b25d942cb07098f; no voters:
I20260619 10:21:44.099628 22678 leader_election.cc:288] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:44.099653 22683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:44.099797 22683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [term 1 LEADER]: Becoming Leader. State: Replica: 60dcf120fb9142ed8b25d942cb07098f, State: Running, Role: LEADER
I20260619 10:21:44.099879 22678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:44.099889 22683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } }
I20260619 10:21:44.100256 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60dcf120fb9142ed8b25d942cb07098f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } } }
I20260619 10:21:44.100353 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:44.100461 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dcf120fb9142ed8b25d942cb07098f. Latest consensus state: current_term: 1 leader_uuid: "60dcf120fb9142ed8b25d942cb07098f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dcf120fb9142ed8b25d942cb07098f" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45837 } } }
I20260619 10:21:44.100553 22690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:44.100569 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:44.101460 22690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:44.103185 22690 catalog_manager.cc:1347] Generated new cluster ID: 2d4b3c1bc9684295805f73f7368ed5b9
I20260619 10:21:44.103230 22690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:44.103715 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:44.092616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.19.65.190:45837)
0619 10:21:44.092791 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:44.092798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:44.092820 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:21:44.093231 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:44.093237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:44.093279 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:44.093381 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:44.094152 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:44.095121 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:44.096114 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:44.096491 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:44.096568 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:44.099249 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:44.099252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:44.099252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:44.099261 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:44.102028 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:44.102461 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:44.102464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:44.102464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:44.102561 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:44.102781 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:44.102783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:44.102783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:44.103090 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:44.103258 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:44.103421 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 10:21:44.103541 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260619 10:21:44.104722 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:21:44.122867 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:44.119136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.19.65.190:45837)
0619 10:21:44.119172 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:44.119173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:44.119307 (+ 134us) server_negotiation.cc:407] Connection header received
0619 10:21:44.119370 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:44.119371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:44.119381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:44.119440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:44.119819 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:44.120504 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:44.121451 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:44.121734 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:44.121787 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:44.122428 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:44.122429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:44.122429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:44.122624 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:44.122641 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:44.122647 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:44.122698 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:44.122770 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:21:44.122793 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":17}
I20260619 10:21:44.124461 22690 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:44.125118 22690 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:44.132819 22690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60dcf120fb9142ed8b25d942cb07098f: Generated new TSK 0
I20260619 10:21:44.132989 22690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:21:44.139089 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22610
2026-06-19T10:21:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1288 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:44 dist-test-slave-xd2q krb5kdc[22711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:44 dist-test-slave-xd2q krb5kdc[22711](info): set up 2 sockets
Jun 19 10:21:44 dist-test-slave-xd2q krb5kdc[22711](info): commencing operation
krb5kdc: starting...
W20260619 10:21:45.158969 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:45.266400 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46861
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:45.343541 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:45.343760 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:45.343793 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:45.344880 22724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:45.344954 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:45.344975 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:45.344996 22724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:45.345023 22724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:45.346410 22724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:45.346673 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:45.346880 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:45.351533 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.351625 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.351531 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:45.351557 22724 server_base.cc:1034] running on GCE node
I20260619 10:21:45.351962 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:45.352183 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:45.353356 22724 hybrid_clock.cc:648] HybridClock initialized: now 1781864505353338 us; error 40 us; skew 500 ppm
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:45.355758 22724 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:45.356171 22724 webserver.cc:460] Webserver started at http://127.19.65.190:46871/ using document root <none> and password file <none>
I20260619 10:21:45.356385 22724 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:45.356473 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:45.356616 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:45.357571 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "be9a3fdf4e02418da1f3813a784d001a"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e76dde9b350ae05a3e6d5868f9cb9cc4"
server_key_iv: "78f2de5a40fa3a2dfd908a7a637dce62"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.357914 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "be9a3fdf4e02418da1f3813a784d001a"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e76dde9b350ae05a3e6d5868f9cb9cc4"
server_key_iv: "78f2de5a40fa3a2dfd908a7a637dce62"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.359215 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:45.360044 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.360203 22724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:45.360284 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "be9a3fdf4e02418da1f3813a784d001a"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e76dde9b350ae05a3e6d5868f9cb9cc4"
server_key_iv: "78f2de5a40fa3a2dfd908a7a637dce62"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.360337 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:45.382007 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:45.384186 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:45.384289 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:45.388178 22724 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46861
I20260619 10:21:45.388230 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46861 every 8 connection(s)
I20260619 10:21:45.388574 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:45.389467 22792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:45.390337 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22724
I20260619 10:21:45.390429 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:45.390576 19718 external_mini_cluster.cc:1428] Setting key cd47f4b11f20ca7014477242d3e1b6ee
I20260619 10:21:45.392457 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a: Bootstrap starting.
I20260619 10:21:45.392985 22792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:45.393251 22792 log.cc:824] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:45.393949 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a: No bootstrap required, opened a new log
I20260619 10:21:45.396139 22792 raft_consensus.cc:348] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } }
I20260619 10:21:45.396289 22792 raft_consensus.cc:374] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:45.396338 22792 raft_consensus.cc:729] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9a3fdf4e02418da1f3813a784d001a, State: Initialized, Role: FOLLOWER
I20260619 10:21:45.396486 22792 consensus_queue.cc:260] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } }
I20260619 10:21:45.396580 22792 raft_consensus.cc:388] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:45.396624 22792 raft_consensus.cc:482] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:45.396672 22792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:45.397222 22792 raft_consensus.cc:504] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } }
I20260619 10:21:45.397326 22792 leader_election.cc:302] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be9a3fdf4e02418da1f3813a784d001a; no voters:
I20260619 10:21:45.397498 22792 leader_election.cc:288] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:45.397553 22797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:45.397688 22797 raft_consensus.cc:686] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [term 1 LEADER]: Becoming Leader. State: Replica: be9a3fdf4e02418da1f3813a784d001a, State: Running, Role: LEADER
I20260619 10:21:45.397787 22792 sys_catalog.cc:564] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:45.397811 22797 consensus_queue.cc:237] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } }
I20260619 10:21:45.398226 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be9a3fdf4e02418da1f3813a784d001a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } } }
I20260619 10:21:45.398221 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be9a3fdf4e02418da1f3813a784d001a. Latest consensus state: current_term: 1 leader_uuid: "be9a3fdf4e02418da1f3813a784d001a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a3fdf4e02418da1f3813a784d001a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46861 } } }
I20260619 10:21:45.398327 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:45.398322 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:45.398986 22803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:45.399506 22803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:45.400892 22803 catalog_manager.cc:1347] Generated new cluster ID: 69b105dc5ce9477b90fa0d7bb0bb48ff
I20260619 10:21:45.400947 22803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:45.401368 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.391045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.19.65.190:46861)
0619 10:21:45.391185 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:45.391198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:45.391219 (+ 21us) server_negotiation.cc:407] Connection header received
0619 10:21:45.391637 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:45.391643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:45.391691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:45.391791 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:45.392486 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.393388 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.394226 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.394807 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.394914 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.397145 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:45.397149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:45.397149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:45.397160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:45.399919 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.400281 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.400284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.400285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.400382 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.400612 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.400613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.400613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.400893 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:45.401044 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:45.401105 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 10:21:45.401226 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260619 10:21:45.406250 22803 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:45.406901 22803 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:45.414508 22803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be9a3fdf4e02418da1f3813a784d001a: Generated new TSK 0
I20260619 10:21:45.414641 22803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:45.438127 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:45.518054 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:45.518267 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:45.518302 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:45.519467 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:45.519524 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:45.519567 22818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:45.520946 22818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:45.521186 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:45.521400 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:45.526788 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.526839 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.526839 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:45.527149 22818 server_base.cc:1034] running on GCE node
I20260619 10:21:45.527427 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:45.527693 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:45.528838 22818 hybrid_clock.cc:648] HybridClock initialized: now 1781864505528811 us; error 40 us; skew 500 ppm
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:45.531430 22818 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:45.531879 22818 webserver.cc:460] Webserver started at http://127.19.65.129:36819/ using document root <none> and password file <none>
I20260619 10:21:45.532162 22818 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:45.532235 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:45.532406 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:45.533464 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "b45b52a54e0e47c3a23a91e625098802"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d7dd7eb5119ae36b61b5cef4bb70c8bd"
server_key_iv: "182230f33f9a60fccd96b892bb2c0661"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.533849 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "b45b52a54e0e47c3a23a91e625098802"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d7dd7eb5119ae36b61b5cef4bb70c8bd"
server_key_iv: "182230f33f9a60fccd96b892bb2c0661"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.535205 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:45.536002 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.536173 22818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:45.536268 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "b45b52a54e0e47c3a23a91e625098802"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d7dd7eb5119ae36b61b5cef4bb70c8bd"
server_key_iv: "182230f33f9a60fccd96b892bb2c0661"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.536361 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:45.559561 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:45.561311 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:45.561506 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:45.561756 22818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:45.562120 22818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:45.562184 22818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.562219 22818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:45.562251 22818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.568670 22818 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40375
I20260619 10:21:45.568707 22946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40375 every 8 connection(s)
I20260619 10:21:45.569021 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:45.573130 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22818
I20260619 10:21:45.573225 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:45.573403 19718 external_mini_cluster.cc:1428] Setting key fdf7549f3bb0c9414b9fe4de915ae297
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:45.578346 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.569787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:54105 (local address 127.19.65.190:46861)
0619 10:21:45.569854 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:45.569857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:45.570073 (+ 216us) server_negotiation.cc:407] Connection header received
0619 10:21:45.570560 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:45.570561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:45.570571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:45.570620 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:45.571555 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.572028 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.573017 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.573331 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.573403 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.575661 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:45.575663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:45.575663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:45.575670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:45.577045 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.577448 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.577449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.577449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.577504 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.577748 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.577748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.577749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.577853 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:45.577924 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:45.578183 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 10:21:45.578263 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:21:45.578509 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.569770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46861 (local address 127.19.65.129:54105)
0619 10:21:45.569961 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:45.569995 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:45.570481 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:45.570671 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:45.570675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:45.571255 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:45.571473 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.571478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.572109 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.572110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.572939 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.572941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.573524 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.573526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.573689 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.574204 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:45.574219 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:45.575565 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:45.577143 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.577149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.577151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.577360 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.577572 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.577573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.577573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.577679 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.577927 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:45.577933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:45.578060 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 10:21:45.578162 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 10:21:45.578299 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260619 10:21:45.579128 22947 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46861
I20260619 10:21:45.579222 22947 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:45.579409 22947 heartbeater.cc:511] Master 127.19.65.190:46861 requested a full tablet report, sending...
I20260619 10:21:45.579942 22756 ts_manager.cc:194] Registered new tserver with Master: b45b52a54e0e47c3a23a91e625098802 (127.19.65.129:40375)
I20260619 10:21:45.580524 22756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:54105
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:45.596055 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:45.674965 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:45.675161 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:45.675182 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:45.676331 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:45.676371 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:45.676430 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:45.677819 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:45.678052 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:45.678256 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:45.682809 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.682821 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.682821 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:45.683059 22952 server_base.cc:1034] running on GCE node
I20260619 10:21:45.683256 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:45.683525 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:45.684684 22952 hybrid_clock.cc:648] HybridClock initialized: now 1781864505684655 us; error 42 us; skew 500 ppm
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:45.687036 22952 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:45.687492 22952 webserver.cc:460] Webserver started at http://127.19.65.130:45857/ using document root <none> and password file <none>
I20260619 10:21:45.687667 22952 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:45.687705 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:45.687783 22952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:45.688657 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "9907cdc88f384673882a86466043e06c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e7c52382da229af40a9fe1bbf3351e42"
server_key_iv: "57dd9cae7c0feb46038d3e02470af252"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.688941 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "9907cdc88f384673882a86466043e06c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e7c52382da229af40a9fe1bbf3351e42"
server_key_iv: "57dd9cae7c0feb46038d3e02470af252"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.690095 22952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:21:45.690706 22967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.690896 22952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:45.690982 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "9907cdc88f384673882a86466043e06c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "e7c52382da229af40a9fe1bbf3351e42"
server_key_iv: "57dd9cae7c0feb46038d3e02470af252"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.691031 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:45.714432 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:45.716578 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:45.716681 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:45.716845 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:45.717058 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:45.717085 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.717108 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:45.717120 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.722348 22952 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:45541
I20260619 10:21:45.722389 23080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:45541 every 8 connection(s)
I20260619 10:21:45.722715 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:45.730085 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 22952
I20260619 10:21:45.730187 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:45.730372 19718 external_mini_cluster.cc:1428] Setting key cdef09a8f008b0de20b5cb91d91f3468
I20260619 10:21:45.730988 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.723352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:36141 (local address 127.19.65.190:46861)
0619 10:21:45.723392 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:45.723395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:45.723592 (+ 197us) server_negotiation.cc:407] Connection header received
0619 10:21:45.723997 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:45.723998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:45.724007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:45.724055 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:45.724838 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.725342 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.726268 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.726445 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.726494 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.728382 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:45.728383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:45.728383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:45.728389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:45.729488 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.729806 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.729807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.729807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.729852 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.730131 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.730131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.730132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.730273 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:45.730329 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:45.730808 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 10:21:45.730898 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:45.731123 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.723320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46861 (local address 127.19.65.130:36141)
0619 10:21:45.723487 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:45.723522 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:45.723927 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:45.724101 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:45.724104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:45.724607 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:45.724771 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.724775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.725430 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.725431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.726200 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.726202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.726512 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.726513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.726607 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.726975 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:45.726982 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:45.728304 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:45.729553 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.729557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.729559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.729748 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.729930 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.729931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.729932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.730065 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.730327 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:45.730332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:45.730506 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 10:21:45.730750 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 10:21:45.730898 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260619 10:21:45.731598 23081 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46861
I20260619 10:21:45.731694 23081 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:45.731874 23081 heartbeater.cc:511] Master 127.19.65.190:46861 requested a full tablet report, sending...
I20260619 10:21:45.732211 22756 ts_manager.cc:194] Registered new tserver with Master: 9907cdc88f384673882a86466043e06c (127.19.65.130:45541)
I20260619 10:21:45.732693 22756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:36141
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:45.753407 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:45.831545 23086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:45.831756 23086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:45.831796 23086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:45.832938 23086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:45.833009 23086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:45.833055 23086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:45.834460 23086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:45.834698 23086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:45.834941 23086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:45.839721 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:45.839870 23086 server_base.cc:1034] running on GCE node
W20260619 10:21:45.839745 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:45.839743 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:45.840126 23086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:45.840319 23086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:45.841480 23086 hybrid_clock.cc:648] HybridClock initialized: now 1781864505841462 us; error 30 us; skew 500 ppm
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:45.843940 23086 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:45.844424 23086 webserver.cc:460] Webserver started at http://127.19.65.131:44651/ using document root <none> and password file <none>
I20260619 10:21:45.844650 23086 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:45.844713 23086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:45.844858 23086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:45.845815 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "5d84f626bb744480a2ef059d4da1166c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d39d1e6496044af801a3997f9a86e04b"
server_key_iv: "67c6eb05a58fa295f66a0f1baa769bca"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.846163 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "5d84f626bb744480a2ef059d4da1166c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d39d1e6496044af801a3997f9a86e04b"
server_key_iv: "67c6eb05a58fa295f66a0f1baa769bca"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.847445 23086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:45.848127 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.848295 23086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:45.848392 23086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "5d84f626bb744480a2ef059d4da1166c"
format_stamp: "Formatted at 2026-06-19 10:21:45 on dist-test-slave-xd2q"
server_key: "d39d1e6496044af801a3997f9a86e04b"
server_key_iv: "67c6eb05a58fa295f66a0f1baa769bca"
server_key_version: "encryptionkey@0"
I20260619 10:21:45.848475 23086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:45.866918 23086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:45.868569 23086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:45.868711 23086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:45.868904 23086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:45.869189 23086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:45.869235 23086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.869264 23086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:45.869290 23086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:45.874569 23086 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:37361
I20260619 10:21:45.874636 23214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:37361 every 8 connection(s)
I20260619 10:21:45.874917 23086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:45.877225 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23086
I20260619 10:21:45.877326 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:45.877502 19718 external_mini_cluster.cc:1428] Setting key f9b7344ebc2e60d22b89b355b0acca61
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:45.883318 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.875635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43417 (local address 127.19.65.190:46861)
0619 10:21:45.875675 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:45.875677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:45.875853 (+ 176us) server_negotiation.cc:407] Connection header received
0619 10:21:45.876353 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:45.876355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:45.876367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:45.876429 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:45.877327 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.878004 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.879037 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.879215 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.879261 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.881077 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:45.881078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:45.881078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:45.881084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:45.882363 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.882660 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.882661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.882661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.882704 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.882889 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.882889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.882889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.882971 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:45.883014 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:45.883187 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 10:21:45.883234 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:21:45.883548 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.875607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46861 (local address 127.19.65.131:43417)
0619 10:21:45.875737 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:45.875777 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:45.876246 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:45.876489 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:45.876493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:45.877051 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:45.877234 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.877238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.878099 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.878099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.878963 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:45.878966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.879277 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.879277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:45.879447 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.879803 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:45.879810 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:45.880998 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:45.882429 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.882433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.882434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.882605 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.882753 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:45.882753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:45.882753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:45.882839 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:45.883018 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:45.883021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:45.883110 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 10:21:45.883177 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:21:45.883272 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260619 10:21:45.883962 23215 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46861
I20260619 10:21:45.884053 23215 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:45.884212 23215 heartbeater.cc:511] Master 127.19.65.190:46861 requested a full tablet report, sending...
I20260619 10:21:45.884583 22756 ts_manager.cc:194] Registered new tserver with Master: 5d84f626bb744480a2ef059d4da1166c (127.19.65.131:37361)
I20260619 10:21:45.884950 22756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:43417
I20260619 10:21:45.888870 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864505, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:45.909922 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:45.903243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.19.65.190:46861)
0619 10:21:45.903356 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:45.903358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:45.903409 (+ 51us) server_negotiation.cc:407] Connection header received
0619 10:21:45.903479 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:45.903480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:45.903491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:45.903552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:45.904094 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.904554 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.905242 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:45.905436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:45.905509 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:45.907651 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:45.907653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:45.907653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:45.907662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:45.908815 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.909151 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.909152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.909152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.909207 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:45.909395 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:45.909396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:45.909396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:45.909519 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:45.909643 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:45.909739 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 10:21:45.909805 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 10:21:45.910152 22756 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52340
I20260619 10:21:45.910564 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22818
I20260619 10:21:45.914745 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22952
I20260619 10:21:45.918690 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23086
I20260619 10:21:45.922627 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 22724
2026-06-19T10:21:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1785 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[23227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[23227](info): set up 2 sockets
Jun 19 10:21:45 dist-test-slave-xd2q krb5kdc[23227](info): commencing operation
krb5kdc: starting...
W20260619 10:21:46.935103 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:47.046514 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36773
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:47.122391 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:47.122575 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:47.122596 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:47.123637 23240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:47.123673 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:47.123685 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:47.123696 23240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:47.123708 23240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:47.125138 23240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:47.125362 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:47.125594 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:47.130225 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.130225 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.130245 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:47.130249 23240 server_base.cc:1034] running on GCE node
I20260619 10:21:47.130664 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:47.130972 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:47.132138 23240 hybrid_clock.cc:648] HybridClock initialized: now 1781864507132116 us; error 46 us; skew 500 ppm
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:47.137931 23240 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:47.138381 23240 webserver.cc:460] Webserver started at http://127.19.65.190:42987/ using document root <none> and password file <none>
I20260619 10:21:47.138584 23240 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:47.138646 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:47.138787 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:47.139779 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "016f404de9614158865458763317946b"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "0564f46a64194bba4fce2005af83a126"
server_key_iv: "78f3aff94928f3a8d844bb460e7e5f68"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.140138 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "016f404de9614158865458763317946b"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "0564f46a64194bba4fce2005af83a126"
server_key_iv: "78f3aff94928f3a8d844bb460e7e5f68"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.141441 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:47.142141 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.142360 23240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:47.142462 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "016f404de9614158865458763317946b"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "0564f46a64194bba4fce2005af83a126"
server_key_iv: "78f3aff94928f3a8d844bb460e7e5f68"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.142537 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:47.158181 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:47.159942 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:47.160089 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:47.163694 23240 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36773
I20260619 10:21:47.163743 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36773 every 8 connection(s)
I20260619 10:21:47.164233 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:47.164886 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:47.167093 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b: Bootstrap starting.
I20260619 10:21:47.167704 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:47.168020 23308 log.cc:824] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:47.168651 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b: No bootstrap required, opened a new log
I20260619 10:21:47.170135 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } }
I20260619 10:21:47.170280 23308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:47.170317 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016f404de9614158865458763317946b, State: Initialized, Role: FOLLOWER
I20260619 10:21:47.170445 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23240
I20260619 10:21:47.170451 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } }
I20260619 10:21:47.170523 23308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:47.170535 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:47.170562 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:47.170610 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:47.170706 19718 external_mini_cluster.cc:1428] Setting key 2f4ede404e33619065e40a2f85a98b0c
I20260619 10:21:47.171595 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } }
I20260619 10:21:47.171725 23308 leader_election.cc:302] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 016f404de9614158865458763317946b; no voters:
I20260619 10:21:47.171929 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:47.171955 23313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:47.172063 23313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [term 1 LEADER]: Becoming Leader. State: Replica: 016f404de9614158865458763317946b, State: Running, Role: LEADER
I20260619 10:21:47.172194 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:47.172206 23313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } }
I20260619 10:21:47.172883 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "016f404de9614158865458763317946b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } } }
I20260619 10:21:47.172972 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:47.173184 23322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:47.173189 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 016f404de9614158865458763317946b. Latest consensus state: current_term: 1 leader_uuid: "016f404de9614158865458763317946b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016f404de9614158865458763317946b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36773 } } }
I20260619 10:21:47.173324 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:47.173795 23322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:47.175401 23322 catalog_manager.cc:1347] Generated new cluster ID: e9161cb18c1645e1926203718e2064fc
I20260619 10:21:47.175452 23322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:47.181766 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.171140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.19.65.190:36773)
0619 10:21:47.171353 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.171368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.171392 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:21:47.171473 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.171477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.171513 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.171636 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.172444 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.173767 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.174715 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.175347 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.175454 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.178204 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.178208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.178208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.178219 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.180175 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.180551 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.180553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.180554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.180679 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.180873 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.180873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.180874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.181160 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.181335 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.181448 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.181580 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260619 10:21:47.182300 23322 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:47.182886 23322 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:47.186743 23322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 016f404de9614158865458763317946b: Generated new TSK 0
I20260619 10:21:47.186897 23322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:47.214677 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:47.292533 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:47.292737 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:47.292769 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:47.293849 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:47.293913 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:47.293953 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:47.295495 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:47.295738 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:47.295961 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:47.301352 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.301498 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.301350 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:47.301566 23334 server_base.cc:1034] running on GCE node
I20260619 10:21:47.301826 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:47.302063 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:47.303229 23334 hybrid_clock.cc:648] HybridClock initialized: now 1781864507303208 us; error 38 us; skew 500 ppm
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:47.305709 23334 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:47.306058 23334 webserver.cc:460] Webserver started at http://127.19.65.129:45211/ using document root <none> and password file <none>
I20260619 10:21:47.306221 23334 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:47.306259 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:47.306342 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:47.307269 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "fef318d06f2f422d93e70e38c5022609"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "5e8869b81726b97dcfb2909247d7ca6e"
server_key_iv: "d9e1800c4d85504595309c47e93cc2a4"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.307626 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "fef318d06f2f422d93e70e38c5022609"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "5e8869b81726b97dcfb2909247d7ca6e"
server_key_iv: "d9e1800c4d85504595309c47e93cc2a4"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.308779 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:47.309417 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.309602 23334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:47.309681 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "fef318d06f2f422d93e70e38c5022609"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "5e8869b81726b97dcfb2909247d7ca6e"
server_key_iv: "d9e1800c4d85504595309c47e93cc2a4"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.309732 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:47.327555 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:47.329223 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:47.329416 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:47.329587 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:47.329811 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:47.329840 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.329859 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:47.329872 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.335254 23334 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:41275
I20260619 10:21:47.335337 23462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:41275 every 8 connection(s)
I20260619 10:21:47.335636 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:47.339545 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23334
I20260619 10:21:47.339658 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:47.339881 19718 external_mini_cluster.cc:1428] Setting key 74a243923d0c9357e598bab86dfde044
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:47.345752 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.336323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36773 (local address 127.19.65.129:37679)
0619 10:21:47.336472 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:47.336504 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:47.337339 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:47.337597 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:47.337600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:47.337828 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:47.337997 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.338001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.338708 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.338709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.339548 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.339550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.340077 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.340078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.340219 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.340753 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:47.340763 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:47.342363 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:47.344337 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.344342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.344344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.344636 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.344866 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.344867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.344867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.344961 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.345255 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:47.345259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:47.345362 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 10:21:47.345450 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 10:21:47.345559 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260619 10:21:47.345781 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.336380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37679 (local address 127.19.65.190:36773)
0619 10:21:47.336456 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.336458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.336584 (+ 126us) server_negotiation.cc:407] Connection header received
0619 10:21:47.337453 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.337454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.337469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.337532 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.338071 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.338583 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.339634 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.339970 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.340055 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.342482 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.342483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.342484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.342492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.344256 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.344711 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.344712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.344713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.344782 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.345031 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.345032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.345033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.345179 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.345270 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.345508 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.345608 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 10:21:47.346619 23463 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36773
I20260619 10:21:47.346707 23463 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:47.346879 23463 heartbeater.cc:511] Master 127.19.65.190:36773 requested a full tablet report, sending...
I20260619 10:21:47.347515 23272 ts_manager.cc:194] Registered new tserver with Master: fef318d06f2f422d93e70e38c5022609 (127.19.65.129:41275)
I20260619 10:21:47.348438 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:37679
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:47.366171 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:47.444011 23468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:47.444199 23468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:47.444221 23468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:47.445426 23468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:47.445466 23468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:47.445492 23468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:47.446998 23468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:47.447182 23468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:47.447384 23468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:21:47.452086 23468 server_base.cc:1034] running on GCE node
W20260619 10:21:47.452061 23476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.452091 23473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.452061 23474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:47.452471 23468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:47.452653 23468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:47.453817 23468 hybrid_clock.cc:648] HybridClock initialized: now 1781864507453811 us; error 34 us; skew 500 ppm
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:47.456179 23468 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:47.456619 23468 webserver.cc:460] Webserver started at http://127.19.65.130:41009/ using document root <none> and password file <none>
I20260619 10:21:47.456844 23468 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:47.456912 23468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:47.457069 23468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:47.458027 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "2fbdb3a52d0e4a46b401bf87c814fd82"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "43d62e807f6a206d518e2b6f0ee26641"
server_key_iv: "cecfd3f2639e11d0e6fe9fdd78919a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.458356 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "2fbdb3a52d0e4a46b401bf87c814fd82"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "43d62e807f6a206d518e2b6f0ee26641"
server_key_iv: "cecfd3f2639e11d0e6fe9fdd78919a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.459609 23468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:21:47.460264 23483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.460428 23468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:47.460520 23468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "2fbdb3a52d0e4a46b401bf87c814fd82"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "43d62e807f6a206d518e2b6f0ee26641"
server_key_iv: "cecfd3f2639e11d0e6fe9fdd78919a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.460593 23468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:47.482293 23468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:47.484102 23468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:47.484257 23468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:47.484474 23468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:47.484757 23468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:47.484807 23468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.484856 23468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:47.484885 23468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.490589 23468 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:40063
I20260619 10:21:47.490641 23596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:40063 every 8 connection(s)
I20260619 10:21:47.490924 23468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:47.499625 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.491537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:56673 (local address 127.19.65.190:36773)
0619 10:21:47.491579 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.491581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.491796 (+ 215us) server_negotiation.cc:407] Connection header received
0619 10:21:47.492642 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.492644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.492654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.492709 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.493267 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.493776 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.494683 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.494880 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.494927 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.496934 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.496935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.496935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.496941 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.498245 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.498668 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.498669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.498669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.498725 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.498962 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.498963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.498964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.499069 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.499142 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.499459 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.499538 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:47.499847 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.491516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36773 (local address 127.19.65.130:56673)
0619 10:21:47.491681 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:47.491719 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:47.492545 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:47.492781 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:47.492785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:47.493014 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:47.493187 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.493191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.493851 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.493851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.494616 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.494617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.494945 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.494946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.495069 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.495527 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:47.495534 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:47.496839 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:47.498367 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.498373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.498375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.498581 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.498793 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.498793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.498794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.498898 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.499133 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:47.499139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:47.499270 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 10:21:47.499436 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 10:21:47.499585 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260619 10:21:47.500259 23597 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36773
I20260619 10:21:47.500377 23597 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:47.500545 23597 heartbeater.cc:511] Master 127.19.65.190:36773 requested a full tablet report, sending...
I20260619 10:21:47.500560 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23468
I20260619 10:21:47.500653 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:47.500846 19718 external_mini_cluster.cc:1428] Setting key 69fc04aa55400a477ba4014524c84c6b
I20260619 10:21:47.500972 23272 ts_manager.cc:194] Registered new tserver with Master: 2fbdb3a52d0e4a46b401bf87c814fd82 (127.19.65.130:40063)
I20260619 10:21:47.501564 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:56673
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:47.524086 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:47.600713 23602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:47.600893 23602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:47.600914 23602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:47.602020 23602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:47.602059 23602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:47.602095 23602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:47.603578 23602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:47.603782 23602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:47.603987 23602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:47.608659 23607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.608690 23608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:47.608664 23610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:47.608947 23602 server_base.cc:1034] running on GCE node
I20260619 10:21:47.609134 23602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:47.609321 23602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:47.610476 23602 hybrid_clock.cc:648] HybridClock initialized: now 1781864507610464 us; error 24 us; skew 500 ppm
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:47.612983 23602 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:47.613384 23602 webserver.cc:460] Webserver started at http://127.19.65.131:39217/ using document root <none> and password file <none>
I20260619 10:21:47.613603 23602 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:47.613664 23602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:47.613803 23602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:47.614734 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "52e4f1f32f7841849bc3540446ffe205"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "8568ae9fdb50c06c33901f184c6d3187"
server_key_iv: "e7cb204feca3217e5ea53a91131374c8"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.615097 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "52e4f1f32f7841849bc3540446ffe205"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "8568ae9fdb50c06c33901f184c6d3187"
server_key_iv: "e7cb204feca3217e5ea53a91131374c8"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.616501 23602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:47.617321 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.617524 23602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:47.617610 23602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "52e4f1f32f7841849bc3540446ffe205"
format_stamp: "Formatted at 2026-06-19 10:21:47 on dist-test-slave-xd2q"
server_key: "8568ae9fdb50c06c33901f184c6d3187"
server_key_iv: "e7cb204feca3217e5ea53a91131374c8"
server_key_version: "encryptionkey@0"
I20260619 10:21:47.617688 23602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:47.666709 23602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:47.669555 23602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:47.669734 23602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:47.669989 23602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:47.670317 23602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:47.670370 23602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.670413 23602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:47.670444 23602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:47.677106 23602 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40809
I20260619 10:21:47.677145 23730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:40809 every 8 connection(s)
I20260619 10:21:47.677487 23602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:47.678282 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23602
I20260619 10:21:47.678399 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:47.678579 19718 external_mini_cluster.cc:1428] Setting key af4284b5f17aea4619ba353266471bad
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:47.686527 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.678178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40187 (local address 127.19.65.190:36773)
0619 10:21:47.678247 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.678249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.678484 (+ 235us) server_negotiation.cc:407] Connection header received
0619 10:21:47.679689 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.679689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.679697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.679741 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.680279 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.680730 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.681695 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.681895 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.681942 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.684066 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.684068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.684068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.684074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.685392 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.685771 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.685772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.685772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.685818 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.686028 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.686028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.686028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.686120 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.686166 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.686413 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.686466 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:21:47.686793 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.678178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36773 (local address 127.19.65.131:40187)
0619 10:21:47.678349 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:47.678388 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:47.679603 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:47.679794 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:47.679797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:47.680028 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:47.680208 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.680211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.680806 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.680807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.681577 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:47.681579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.681977 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.681978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:47.682081 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.682626 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:47.682633 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:47.683986 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:47.685470 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.685476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.685477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.685694 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.685876 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:47.685876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:47.685877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:47.685973 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:47.686175 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:47.686179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:47.686297 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 10:21:47.686385 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 10:21:47.686498 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260619 10:21:47.687398 23731 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36773
I20260619 10:21:47.687474 23731 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:47.687652 23731 heartbeater.cc:511] Master 127.19.65.190:36773 requested a full tablet report, sending...
I20260619 10:21:47.688052 23272 ts_manager.cc:194] Registered new tserver with Master: 52e4f1f32f7841849bc3540446ffe205 (127.19.65.131:40809)
I20260619 10:21:47.688427 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:40187
I20260619 10:21:47.690193 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:47.711777 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.704317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.19.65.129:41275)
0619 10:21:47.704473 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.704478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.704496 (+ 18us) server_negotiation.cc:407] Connection header received
0619 10:21:47.704517 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.704519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.704531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.704593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.705146 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.705844 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.706609 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.706906 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.706960 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.709259 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.709266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.709267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.709273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.710581 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.710964 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.710966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.710966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.711018 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.711193 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.711194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.711194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.711349 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.711466 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.711575 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.711705 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:47.719540 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:47.712972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.19.65.190:36773)
0619 10:21:47.713042 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:47.713044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:47.713055 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:47.713168 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:47.713169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:47.713179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:47.713267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:47.713773 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.714269 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.714955 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:47.715172 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:47.715270 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:47.717416 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:47.717417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:47.717417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:47.717423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:47.718563 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.718838 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.718839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.718839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.718883 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:47.719027 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:47.719027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:47.719027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:47.719134 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:47.719196 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:47.719342 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 10:21:47.719412 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 10:21:47.719748 23272 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47502
I20260619 10:21:47.720064 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23334
I20260619 10:21:47.725030 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23468
I20260619 10:21:47.729046 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23602
I20260619 10:21:47.733034 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23240
2026-06-19T10:21:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1810 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23743](info): set up 2 sockets
Jun 19 10:21:47 dist-test-slave-xd2q krb5kdc[23743](info): commencing operation
krb5kdc: starting...
W20260619 10:21:48.747097 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:48 dist-test-slave-xd2q krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:48.856284 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42995
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:48.933099 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:48.933274 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:48.933293 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:48.934252 23756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:48.934281 23756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:48.934293 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:48.934305 23756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:48.934315 23756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:48.935779 23756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:48.935962 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:48.936112 23756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:48.940673 23764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:48.940651 23761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:48.940644 23762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:48.940943 23756 server_base.cc:1034] running on GCE node
I20260619 10:21:48.941197 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:48.941421 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:48.942586 23756 hybrid_clock.cc:648] HybridClock initialized: now 1781864508942580 us; error 33 us; skew 500 ppm
Jun 19 10:21:48 dist-test-slave-xd2q krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864508, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:48.944921 23756 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:48.945247 23756 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:48.945294 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:48.945377 23756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:48.946285 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "bc09b3ef221f4a53b21a9099234fe17b"
format_stamp: "Formatted at 2026-06-19 10:21:48 on dist-test-slave-xd2q"
server_key: "92da89debe98cf8c6362d55ebbd63f7e"
server_key_iv: "560320e726de4a054fba5f6e1fb4fbe7"
server_key_version: "encryptionkey@0"
I20260619 10:21:48.946593 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "bc09b3ef221f4a53b21a9099234fe17b"
format_stamp: "Formatted at 2026-06-19 10:21:48 on dist-test-slave-xd2q"
server_key: "92da89debe98cf8c6362d55ebbd63f7e"
server_key_iv: "560320e726de4a054fba5f6e1fb4fbe7"
server_key_version: "encryptionkey@0"
I20260619 10:21:48.947817 23756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:48.948457 23770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:48.948644 23756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:48.948727 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "bc09b3ef221f4a53b21a9099234fe17b"
format_stamp: "Formatted at 2026-06-19 10:21:48 on dist-test-slave-xd2q"
server_key: "92da89debe98cf8c6362d55ebbd63f7e"
server_key_iv: "560320e726de4a054fba5f6e1fb4fbe7"
server_key_version: "encryptionkey@0"
I20260619 10:21:48.948778 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:48.956559 23756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:48.958091 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:48.958187 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:48.961913 23756 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42995
I20260619 10:21:48.961936 23822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:42995 every 8 connection(s)
I20260619 10:21:48.962206 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:48.963035 23823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:48.965118 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: Bootstrap starting.
I20260619 10:21:48.965654 23823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:48.965955 23823 log.cc:824] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:48.966559 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: No bootstrap required, opened a new log
I20260619 10:21:48.968148 23823 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } }
I20260619 10:21:48.968302 23823 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:48.968382 23823 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc09b3ef221f4a53b21a9099234fe17b, State: Initialized, Role: FOLLOWER
I20260619 10:21:48.968570 23823 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } }
I20260619 10:21:48.968645 23823 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:48.968685 23823 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:48.968729 23823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:48.969444 23823 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } }
I20260619 10:21:48.969551 23823 leader_election.cc:302] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc09b3ef221f4a53b21a9099234fe17b; no voters:
I20260619 10:21:48.969722 23823 leader_election.cc:288] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:48.969755 23826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:48.969928 23826 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [term 1 LEADER]: Becoming Leader. State: Replica: bc09b3ef221f4a53b21a9099234fe17b, State: Running, Role: LEADER
I20260619 10:21:48.969947 23823 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:48.970018 23826 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } }
I20260619 10:21:48.970237 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23756
I20260619 10:21:48.970322 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:48.970494 19718 external_mini_cluster.cc:1428] Setting key b8f0a3f494b2e5a64948ff7491fc1554
I20260619 10:21:48.971064 23827 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc09b3ef221f4a53b21a9099234fe17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } } }
I20260619 10:21:48.971467 23827 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:48.971236 23828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc09b3ef221f4a53b21a9099234fe17b. Latest consensus state: current_term: 1 leader_uuid: "bc09b3ef221f4a53b21a9099234fe17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc09b3ef221f4a53b21a9099234fe17b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42995 } } }
I20260619 10:21:48.971627 23828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b [sys.catalog]: This master's current role is: LEADER
W20260619 10:21:48.971992 23842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:48.972038 23842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:48.972124 23844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:48.972278 23844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:48.974224 23844 catalog_manager.cc:1347] Generated new cluster ID: db14fd10c4c443038b6446d16060a006
I20260619 10:21:48.974275 23844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:21:48 dist-test-slave-xd2q krb5kdc[23743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:48.980418 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:48.970923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.19.65.190:42995)
0619 10:21:48.971227 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:48.971235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:48.971265 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:21:48.971447 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:48.971455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:48.971506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:48.972440 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:48.972524 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:48.973812 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:48.974685 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:48.975139 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:48.975256 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:48.977450 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:48.977454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:48.977454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:48.977464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:48.978979 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:48.979373 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:48.979375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:48.979376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:48.979456 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:48.979655 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:48.979656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:48.979656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:48.979875 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:48.980031 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:48.980126 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:21:48.980248 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":66,"threads_started":1}
I20260619 10:21:48.996400 23844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:48.997118 23844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:49.010273 23844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: Generated new TSK 0
I20260619 10:21:49.010452 23844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:49.039844 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:49.118244 23849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:49.118419 23849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:49.118441 23849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:49.119582 23849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:49.119621 23849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:49.119647 23849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:49.121127 23849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:49.121306 23849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:49.121454 23849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:49.126217 23855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.126271 23854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.126303 23857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:49.126451 23849 server_base.cc:1034] running on GCE node
I20260619 10:21:49.126652 23849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:49.126873 23849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:49.128027 23849 hybrid_clock.cc:648] HybridClock initialized: now 1781864509128025 us; error 37 us; skew 500 ppm
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:49.130543 23849 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:49.130870 23849 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:49.130918 23849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:49.131007 23849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:49.131974 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "a1c59aaecdbf4469a5abee6ec46d4d09"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "63d4850094ca4c37dddf799bc1d110c7"
server_key_iv: "fc056c7f93d80a765c62079332021018"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.132265 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c59aaecdbf4469a5abee6ec46d4d09"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "63d4850094ca4c37dddf799bc1d110c7"
server_key_iv: "fc056c7f93d80a765c62079332021018"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.133407 23849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:21:49.134037 23863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.134215 23849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:49.134299 23849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "a1c59aaecdbf4469a5abee6ec46d4d09"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "63d4850094ca4c37dddf799bc1d110c7"
server_key_iv: "fc056c7f93d80a765c62079332021018"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.134393 23849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:49.158851 23849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:49.161046 23849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:49.161202 23849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:49.161373 23849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:49.161612 23849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:49.161638 23849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.161657 23849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:49.161669 23849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.167397 23849 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:35785
I20260619 10:21:49.167443 23976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:35785 every 8 connection(s)
I20260619 10:21:49.167681 23849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:49.174995 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23849
I20260619 10:21:49.175113 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:49.175354 19718 external_mini_cluster.cc:1428] Setting key 49feaf2abee0661df7f553b1ebfb3aed
I20260619 10:21:49.176435 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.168313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36167 (local address 127.19.65.190:42995)
0619 10:21:49.168385 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.168387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.168567 (+ 180us) server_negotiation.cc:407] Connection header received
0619 10:21:49.168981 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.168982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.168992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.169042 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:49.169966 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.170489 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.171624 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.171799 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.171849 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.173743 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:49.173744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:49.173744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:49.173751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:49.174904 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.175430 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.175431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.175432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.175495 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.175784 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.175785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.175786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.175914 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:49.175988 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.176302 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.176373 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 10:21:49.176710 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.168311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42995 (local address 127.19.65.129:36167)
0619 10:21:49.168445 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.168482 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.168907 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.169098 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.169101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.169352 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:49.169878 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.169884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.170589 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.170590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.171558 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.171561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.171865 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.171865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.171955 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.172429 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:49.172436 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:49.173672 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:49.174992 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.174998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.175018 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.175346 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.175561 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.175561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.175562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.175709 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.175991 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:49.175997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:49.176161 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.176277 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.176447 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260619 10:21:49.177759 23977 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42995
I20260619 10:21:49.177874 23977 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:49.178050 23977 heartbeater.cc:511] Master 127.19.65.190:42995 requested a full tablet report, sending...
I20260619 10:21:49.178535 23787 ts_manager.cc:194] Registered new tserver with Master: a1c59aaecdbf4469a5abee6ec46d4d09 (127.19.65.129:35785)
I20260619 10:21:49.179160 23787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:36167
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:49.197873 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:49.275051 23982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:49.275261 23982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:49.275297 23982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:49.276440 23982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:49.276505 23982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:49.276549 23982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:49.277980 23982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:49.278210 23982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:49.278416 23982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:49.283046 23987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.283070 23990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.283070 23988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:49.283257 23982 server_base.cc:1034] running on GCE node
I20260619 10:21:49.283473 23982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:49.283655 23982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:49.284788 23982 hybrid_clock.cc:648] HybridClock initialized: now 1781864509284780 us; error 33 us; skew 500 ppm
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:49.287158 23982 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:49.287485 23982 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:49.287532 23982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:49.287638 23982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:49.288590 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "426a5971e7a94eeaa75247a89fe0c31c"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "0fe518494674b9f5980270c111ae1571"
server_key_iv: "328d1f93efaa6adecc938da305d79216"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.288892 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "426a5971e7a94eeaa75247a89fe0c31c"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "0fe518494674b9f5980270c111ae1571"
server_key_iv: "328d1f93efaa6adecc938da305d79216"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.290063 23982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:49.290721 23996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.290869 23982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:49.290952 23982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "426a5971e7a94eeaa75247a89fe0c31c"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "0fe518494674b9f5980270c111ae1571"
server_key_iv: "328d1f93efaa6adecc938da305d79216"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.291004 23982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:49.307551 23982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:49.309219 23982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:49.309317 23982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:49.309507 23982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:49.309809 23982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:49.309840 23982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.309859 23982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:49.309906 23982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.315335 23982 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41211
I20260619 10:21:49.315395 24109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:41211 every 8 connection(s)
I20260619 10:21:49.315652 23982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:49.321990 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 23982
I20260619 10:21:49.322104 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:49.322283 19718 external_mini_cluster.cc:1428] Setting key 25cf32636c5e93dfb2285aeb3b843f5b
I20260619 10:21:49.324576 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.316325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:39345 (local address 127.19.65.190:42995)
0619 10:21:49.316366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.316368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.316579 (+ 211us) server_negotiation.cc:407] Connection header received
0619 10:21:49.317005 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.317006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.317014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.317076 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:49.317883 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.318373 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.319330 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.319552 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.319605 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.321571 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:49.321572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:49.321572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:49.321579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:49.323216 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.323637 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.323638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.323639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.323698 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.323945 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.323946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.323946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.324099 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:49.324163 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.324400 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.324490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:49.324713 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.316276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42995 (local address 127.19.65.130:39345)
0619 10:21:49.316456 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.316494 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.316928 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.317131 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.317134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.317362 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:49.317800 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.317804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.318450 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.318451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.319213 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.319215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.319621 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.319622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.319714 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.320117 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:49.320124 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:49.321486 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:49.323329 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.323336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.323338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.323554 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.323760 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.323761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.323761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.323883 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.324159 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:49.324162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:49.324260 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.324364 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.324504 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260619 10:21:49.325166 24110 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42995
I20260619 10:21:49.325258 24110 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:49.325412 24110 heartbeater.cc:511] Master 127.19.65.190:42995 requested a full tablet report, sending...
I20260619 10:21:49.325768 23787 ts_manager.cc:194] Registered new tserver with Master: 426a5971e7a94eeaa75247a89fe0c31c (127.19.65.130:41211)
I20260619 10:21:49.326267 23787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:39345
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:49.345263 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:49.423005 24115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:49.423223 24115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:49.423261 24115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:49.424425 24115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:49.424491 24115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:49.424536 24115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:49.426029 24115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:49.426286 24115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:49.426502 24115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:49.431149 24121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.431164 24120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.431191 24123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:49.431471 24115 server_base.cc:1034] running on GCE node
I20260619 10:21:49.431682 24115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:49.431926 24115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:49.433071 24115 hybrid_clock.cc:648] HybridClock initialized: now 1781864509433058 us; error 43 us; skew 500 ppm
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:49.435585 24115 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:49.435971 24115 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:49.436043 24115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:49.436193 24115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:49.437222 24115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "d0d4742dbc474f49a1ac19f385c066c6"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "91c89725bfd56f1bb183c68143133643"
server_key_iv: "63316c95a8ceeee4e9fd49f4cdfed1fe"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.437637 24115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "d0d4742dbc474f49a1ac19f385c066c6"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "91c89725bfd56f1bb183c68143133643"
server_key_iv: "63316c95a8ceeee4e9fd49f4cdfed1fe"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.438843 24115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:49.439530 24129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.439687 24115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:49.439777 24115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "d0d4742dbc474f49a1ac19f385c066c6"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "91c89725bfd56f1bb183c68143133643"
server_key_iv: "63316c95a8ceeee4e9fd49f4cdfed1fe"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.439860 24115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:49.451421 24115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:49.453606 24115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:49.453753 24115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:49.453970 24115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:49.454257 24115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:49.454310 24115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.454357 24115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:49.454385 24115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.460037 24115 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:39689
I20260619 10:21:49.460107 24242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:39689 every 8 connection(s)
I20260619 10:21:49.460362 24115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:21:49 dist-test-slave-xd2q krb5kdc[23743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864509, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:49.468369 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.461072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48167 (local address 127.19.65.190:42995)
0619 10:21:49.461142 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.461144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.461281 (+ 137us) server_negotiation.cc:407] Connection header received
0619 10:21:49.461675 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.461676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.461687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.461736 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:49.462560 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.463017 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.463956 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.464130 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.464180 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.465947 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:49.465948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:49.465948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:49.465954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:49.467273 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.467664 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.467664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.467665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.467712 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.467909 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.467910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.467910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.467999 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:49.468045 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.468244 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.468298 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:21:49.468488 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.461027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42995 (local address 127.19.65.131:48167)
0619 10:21:49.461170 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.461201 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.461603 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.461781 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.461784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.462024 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:49.462492 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.462496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.463094 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.463095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.463886 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.463888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.464195 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.464196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.464293 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.464646 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:49.464652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:49.465853 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:49.467402 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.467407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.467408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.467591 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.467766 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:49.467766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:49.467767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:49.467858 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:49.468050 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:49.468053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:49.468146 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.468217 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.468314 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260619 10:21:49.468978 24243 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42995
I20260619 10:21:49.469069 24243 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:49.469197 24243 heartbeater.cc:511] Master 127.19.65.190:42995 requested a full tablet report, sending...
I20260619 10:21:49.469591 23787 ts_manager.cc:194] Registered new tserver with Master: d0d4742dbc474f49a1ac19f385c066c6 (127.19.65.131:39689)
I20260619 10:21:49.469610 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24115
I20260619 10:21:49.469710 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:49.469880 19718 external_mini_cluster.cc:1428] Setting key bbe2bd0f95ff45319ba9ecab69391c69
I20260619 10:21:49.470031 23787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:48167
I20260619 10:21:49.470633 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:21:49.477632 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.472100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.19.65.190:42995)
0619 10:21:49.472129 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.472131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.472293 (+ 162us) server_negotiation.cc:407] Connection header received
0619 10:21:49.472362 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.472363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.472372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.472425 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:49.472951 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.473417 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.474126 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.474307 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.474385 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.475497 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:49.475498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:49.475499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:49.475505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:49.476525 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.476870 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.476871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.476871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.476926 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:49.477138 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:49.477139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:49.477139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:49.477264 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:49.477368 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.477468 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.477547 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:21:49.478925 23786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:49.480392 23786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:49.486480 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:39689 (local address 127.0.0.1:52878)
0619 10:21:49.482554 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.482590 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.482677 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.482886 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.482889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.482890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:49.482946 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.482948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.484254 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.484255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.485315 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.485317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.486080 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.486081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.486161 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.486174 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.486248 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.486300 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":58,"threads_started":1}
I20260619 10:21:49.486654 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41211 (local address 127.0.0.1:47024)
0619 10:21:49.482616 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.482624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.482669 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.483017 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.483018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.483020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:49.483086 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.483088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.484039 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.484041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.485246 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.485249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.485902 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.485903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.486486 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.486490 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.486547 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.486585 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2,"thread_start_us":52,"threads_started":1}
I20260619 10:21:49.486418 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:35785 (local address 127.0.0.1:53788)
0619 10:21:49.482554 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.482588 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.482656 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.483017 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.483017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.483019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:49.483067 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.483068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.484351 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.484352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.485139 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.485141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.486121 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.486122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.486185 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.486189 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.486263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.486309 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260619 10:21:49.486740 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.19.65.131:39689)
0619 10:21:49.482474 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.482479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.482651 (+ 172us) server_negotiation.cc:407] Connection header received
0619 10:21:49.482767 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.482769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.482783 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.482882 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:49.483008 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.484155 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.485374 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.486006 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.486068 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.486111 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.486608 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.486658 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
I20260619 10:21:49.486913 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.19.65.129:35785)
0619 10:21:49.482533 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.482536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.482735 (+ 199us) server_negotiation.cc:407] Connection header received
0619 10:21:49.482778 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.482780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.482796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.482870 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:49.483143 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.484262 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.485224 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.485955 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.486027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.486085 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.486456 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.486505 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260619 10:21:49.487088 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.482634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.19.65.130:41211)
0619 10:21:49.482794 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.482798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.482807 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:49.482841 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.482843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.482867 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.482934 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:49.483170 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.483942 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.485338 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.485803 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.485889 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.485959 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.486747 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.487011 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260619 10:21:49.487408 24177 tablet_service.cc:1467] Processing CreateTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 (DEFAULT_TABLE table=test-table [id=ac2d699fa1424cad9e80120346eea64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:49.487526 23911 tablet_service.cc:1467] Processing CreateTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 (DEFAULT_TABLE table=test-table [id=ac2d699fa1424cad9e80120346eea64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:49.487636 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0beb5a845d5444b5b8a4bd01fe6db550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:49.487761 23911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0beb5a845d5444b5b8a4bd01fe6db550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:49.489912 24257 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Bootstrap starting.
I20260619 10:21:49.490353 24258 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Bootstrap starting.
I20260619 10:21:49.490599 24044 tablet_service.cc:1467] Processing CreateTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 (DEFAULT_TABLE table=test-table [id=ac2d699fa1424cad9e80120346eea64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:49.490672 24257 tablet_bootstrap.cc:654] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:49.490875 24044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0beb5a845d5444b5b8a4bd01fe6db550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:49.490993 24258 tablet_bootstrap.cc:654] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:49.491315 24257 log.cc:824] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:49.491626 24258 log.cc:824] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:49.492097 24257 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: No bootstrap required, opened a new log
I20260619 10:21:49.492168 24257 ts_tablet_manager.cc:1397] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:49.492398 24258 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: No bootstrap required, opened a new log
I20260619 10:21:49.492465 24258 ts_tablet_manager.cc:1397] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:49.493907 24261 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Bootstrap starting.
I20260619 10:21:49.494247 24257 raft_consensus.cc:348] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.494431 24257 raft_consensus.cc:374] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:49.494360 24258 raft_consensus.cc:348] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.494465 24257 raft_consensus.cc:729] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d4742dbc474f49a1ac19f385c066c6, State: Initialized, Role: FOLLOWER
I20260619 10:21:49.494496 24261 tablet_bootstrap.cc:654] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:49.494504 24258 raft_consensus.cc:374] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:49.494535 24258 raft_consensus.cc:729] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c59aaecdbf4469a5abee6ec46d4d09, State: Initialized, Role: FOLLOWER
I20260619 10:21:49.494613 24257 consensus_queue.cc:260] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.494648 24258 consensus_queue.cc:260] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.494897 24257 ts_tablet_manager.cc:1428] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:49.494920 24243 heartbeater.cc:503] Master 127.19.65.190:42995 was elected leader, sending a full tablet report...
I20260619 10:21:49.495085 23977 heartbeater.cc:503] Master 127.19.65.190:42995 was elected leader, sending a full tablet report...
I20260619 10:21:49.495103 24261 log.cc:824] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:49.495555 24258 ts_tablet_manager.cc:1428] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:49.495850 24261 tablet_bootstrap.cc:492] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: No bootstrap required, opened a new log
I20260619 10:21:49.495920 24261 ts_tablet_manager.cc:1397] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 10:21:49.497385 24261 raft_consensus.cc:348] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.497529 24261 raft_consensus.cc:374] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:49.497572 24261 raft_consensus.cc:729] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426a5971e7a94eeaa75247a89fe0c31c, State: Initialized, Role: FOLLOWER
I20260619 10:21:49.497676 24261 consensus_queue.cc:260] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.497922 24261 ts_tablet_manager.cc:1428] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:21:49.497977 24110 heartbeater.cc:503] Master 127.19.65.190:42995 was elected leader, sending a full tablet report...
W20260619 10:21:49.566182 24111 tablet.cc:2369] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:49.668319 23978 tablet.cc:2369] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:49.710932 24244 tablet.cc:2369] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:49.729473 24262 raft_consensus.cc:482] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:49.729713 24262 raft_consensus.cc:504] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.730072 24262 leader_election.cc:288] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0d4742dbc474f49a1ac19f385c066c6 (127.19.65.131:39689), 426a5971e7a94eeaa75247a89fe0c31c (127.19.65.130:41211)
I20260619 10:21:49.733067 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.730373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44347 (local address 127.19.65.131:39689)
0619 10:21:49.730428 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.730430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.730447 (+ 17us) server_negotiation.cc:407] Connection header received
0619 10:21:49.730547 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.730549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.730560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.730615 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:49.730817 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.731331 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732342 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.732717 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732772 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.732779 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.732955 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.732998 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:21:49.733098 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.730575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45969 (local address 127.19.65.130:41211)
0619 10:21:49.730644 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.730646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.730658 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:21:49.730684 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.730685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.730694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.730752 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:49.730949 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.731470 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732384 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.732723 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732820 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.732828 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.733002 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.733040 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:21:49.733090 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.730287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:39689 (local address 127.19.65.129:44347)
0619 10:21:49.730396 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.730408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.730461 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.730644 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.730645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.730647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:49.730728 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.730730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.731405 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.731406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.732245 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.732246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.732816 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.732889 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.732892 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.732933 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.732981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":78,"threads_started":1}
I20260619 10:21:49.733109 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.730484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41211 (local address 127.19.65.129:45969)
0619 10:21:49.730585 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.730595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:49.730640 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:49.730772 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:49.730773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:49.730774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:49.730853 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.730856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.731563 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.731564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.732266 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:49.732267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.732837 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.732838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:49.732908 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.732911 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:49.732982 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:21:49.733016 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260619 10:21:49.733516 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0beb5a845d5444b5b8a4bd01fe6db550" candidate_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d4742dbc474f49a1ac19f385c066c6" is_pre_election: true
I20260619 10:21:49.733526 24064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0beb5a845d5444b5b8a4bd01fe6db550" candidate_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426a5971e7a94eeaa75247a89fe0c31c" is_pre_election: true
I20260619 10:21:49.733654 24197 raft_consensus.cc:2454] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c59aaecdbf4469a5abee6ec46d4d09 in term 0.
I20260619 10:21:49.733685 24064 raft_consensus.cc:2454] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c59aaecdbf4469a5abee6ec46d4d09 in term 0.
I20260619 10:21:49.733958 23864 leader_election.cc:302] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426a5971e7a94eeaa75247a89fe0c31c, a1c59aaecdbf4469a5abee6ec46d4d09; no voters:
I20260619 10:21:49.734084 24262 raft_consensus.cc:2781] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:49.734134 24262 raft_consensus.cc:482] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:49.734166 24262 raft_consensus.cc:3037] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:49.734813 24262 raft_consensus.cc:504] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.734934 24262 leader_election.cc:288] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [CANDIDATE]: Term 1 election: Requested vote from peers d0d4742dbc474f49a1ac19f385c066c6 (127.19.65.131:39689), 426a5971e7a94eeaa75247a89fe0c31c (127.19.65.130:41211)
I20260619 10:21:49.735145 24064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0beb5a845d5444b5b8a4bd01fe6db550" candidate_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426a5971e7a94eeaa75247a89fe0c31c"
I20260619 10:21:49.735145 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0beb5a845d5444b5b8a4bd01fe6db550" candidate_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d4742dbc474f49a1ac19f385c066c6"
I20260619 10:21:49.735226 24064 raft_consensus.cc:3037] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:49.735226 24197 raft_consensus.cc:3037] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:49.736163 24197 raft_consensus.cc:2454] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c59aaecdbf4469a5abee6ec46d4d09 in term 1.
I20260619 10:21:49.736166 24064 raft_consensus.cc:2454] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c59aaecdbf4469a5abee6ec46d4d09 in term 1.
I20260619 10:21:49.736354 23864 leader_election.cc:302] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426a5971e7a94eeaa75247a89fe0c31c, a1c59aaecdbf4469a5abee6ec46d4d09; no voters:
I20260619 10:21:49.736464 24262 raft_consensus.cc:2781] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:49.736653 24262 raft_consensus.cc:686] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [term 1 LEADER]: Becoming Leader. State: Replica: a1c59aaecdbf4469a5abee6ec46d4d09, State: Running, Role: LEADER
I20260619 10:21:49.736748 24262 consensus_queue.cc:237] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } }
I20260619 10:21:49.737587 23786 catalog_manager.cc:5495] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1c59aaecdbf4469a5abee6ec46d4d09 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c59aaecdbf4469a5abee6ec46d4d09" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260619 10:21:49.769660 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.767495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.19.65.129:35785)
0619 10:21:49.767551 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.767553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.767560 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:21:49.767637 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.767638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.767653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.767711 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:49.767837 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.768279 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.768958 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:49.769117 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:49.769163 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:49.769313 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:49.769445 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:49.769505 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.769519 (+ 14us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.769533 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:21:49.771481 24197 raft_consensus.cc:1261] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 1 FOLLOWER]: Refusing update from remote peer a1c59aaecdbf4469a5abee6ec46d4d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:49.771498 24064 raft_consensus.cc:1261] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 1 FOLLOWER]: Refusing update from remote peer a1c59aaecdbf4469a5abee6ec46d4d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:49.771797 24268 consensus_queue.cc:1035] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "426a5971e7a94eeaa75247a89fe0c31c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:49.771935 24262 consensus_queue.cc:1035] T 0beb5a845d5444b5b8a4bd01fe6db550 P a1c59aaecdbf4469a5abee6ec46d4d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d4742dbc474f49a1ac19f385c066c6" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:49.775170 24272 mvcc.cc:205] Tried to move back new op lower bound from 7298517032021348352 to 7298517031882391552. Current Snapshot: MvccSnapshot[applied={T|T < 7298517032021348352}]
I20260619 10:21:49.776696 23787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:49.776793 23787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:49.777519 23787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc09b3ef221f4a53b21a9099234fe17b: Sending DeleteTablet for 3 replicas of tablet 0beb5a845d5444b5b8a4bd01fe6db550
I20260619 10:21:49.777788 23911 tablet_service.cc:1514] Processing DeleteTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:49 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:53788
I20260619 10:21:49.777837 24044 tablet_service.cc:1514] Processing DeleteTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:49 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:47024
I20260619 10:21:49.778050 24281 tablet_replica.cc:330] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: stopping tablet replica
I20260619 10:21:49.778057 24177 tablet_service.cc:1514] Processing DeleteTablet for tablet 0beb5a845d5444b5b8a4bd01fe6db550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:49 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52878
I20260619 10:21:49.778214 24281 raft_consensus.cc:2229] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:49.778272 24282 tablet_replica.cc:330] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: stopping tablet replica
I20260619 10:21:49.778374 24282 raft_consensus.cc:2229] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:49.778402 24281 raft_consensus.cc:2258] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:49.778473 24282 raft_consensus.cc:2258] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:49.778537 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23849
I20260619 10:21:49.779596 24282 ts_tablet_manager.cc:1905] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:49.780997 24282 ts_tablet_manager.cc:1918] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:49.781077 24282 log.cc:1196] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/0beb5a845d5444b5b8a4bd01fe6db550
I20260619 10:21:49.781360 24282 ts_tablet_manager.cc:1939] T 0beb5a845d5444b5b8a4bd01fe6db550 P d0d4742dbc474f49a1ac19f385c066c6: Deleting consensus metadata
I20260619 10:21:49.781744 23771 catalog_manager.cc:4841] TS d0d4742dbc474f49a1ac19f385c066c6 (127.19.65.131:39689): tablet 0beb5a845d5444b5b8a4bd01fe6db550 (table test-table [id=ac2d699fa1424cad9e80120346eea64d]) successfully deleted
I20260619 10:21:49.783689 24281 ts_tablet_manager.cc:1905] T 0beb5a845d5444b5b8a4bd01fe6db550 P 426a5971e7a94eeaa75247a89fe0c31c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:21:49.784299 23774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:35785 (error 108)
I20260619 10:21:49.784489 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23982
I20260619 10:21:49.787670 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.787536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:35785 (local address 127.0.0.1:53802)
0619 10:21:49.787591 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.787630 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:35785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 10:21:49.787809 23774 catalog_manager.cc:4584] TS a1c59aaecdbf4469a5abee6ec46d4d09 (127.19.65.129:35785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0beb5a845d5444b5b8a4bd01fe6db550: Network error: Client connection negotiation failed: client connection to 127.19.65.129:35785: connect: Connection refused (error 111)
I20260619 10:21:49.789407 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24115
I20260619 10:21:49.789417 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.789314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41211 (local address 127.0.0.1:47038)
0619 10:21:49.789365 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:49.789390 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.130:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:21:49.794443 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 23756
2026-06-19T10:21:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2061 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T10:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:49Z Disabled control of system clock
I20260619 10:21:49.821260 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42641
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:21:49.898463 24287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:49.898653 24287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:49.898675 24287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:49.899972 24287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:49.900020 24287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:49.900043 24287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:49.900058 24287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:49.900115 24287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:49.901468 24287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:49.901703 24287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:49.902007 24287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:49.906836 24293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:49.906845 24292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:49.906876 24287 server_base.cc:1034] running on GCE node
W20260619 10:21:49.906844 24295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:49.907368 24287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:49.907661 24287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:49.908826 24287 hybrid_clock.cc:648] HybridClock initialized: now 1781864509908814 us; error 27 us; skew 500 ppm
I20260619 10:21:49.910036 24287 webserver.cc:460] Webserver started at http://127.19.65.190:35151/ using document root <none> and password file <none>
I20260619 10:21:49.910280 24287 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:49.910348 24287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:49.910491 24287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:49.911531 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "1bf045bf7066fa389922ee66e273260c"
server_key_iv: "499c67228ec5a5fc701359d4dee268c2"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.911876 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "1bf045bf7066fa389922ee66e273260c"
server_key_iv: "499c67228ec5a5fc701359d4dee268c2"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.913084 24287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:49.913753 24301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:49.914028 24287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:49.914119 24287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62"
format_stamp: "Formatted at 2026-06-19 10:21:49 on dist-test-slave-xd2q"
server_key: "1bf045bf7066fa389922ee66e273260c"
server_key_iv: "499c67228ec5a5fc701359d4dee268c2"
server_key_version: "encryptionkey@0"
I20260619 10:21:49.914186 24287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:49.917369 24287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:49.917615 24287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:49.917752 24287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:49.921299 24287 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42641
I20260619 10:21:49.921332 24353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:42641 every 8 connection(s)
I20260619 10:21:49.921695 24287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:49.922271 24354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:49.924697 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: Bootstrap starting.
I20260619 10:21:49.925016 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24287
I20260619 10:21:49.925118 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:49.925320 19718 external_mini_cluster.cc:1428] Setting key 31da6f955a4cd012b308c44cc8590c26
I20260619 10:21:49.925355 24354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:49.925679 24354 log.cc:824] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:49.926370 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: No bootstrap required, opened a new log
I20260619 10:21:49.927659 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:49.925800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.19.65.190:42641)
0619 10:21:49.926000 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:49.926008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:49.926035 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:21:49.926398 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:49.926426 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:49.926473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:49.926629 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:49.927246 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:49.927250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:49.927250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:49.927255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:49.927330 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:49.927344 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:49.927421 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:49.927458 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:21:49.927510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260619 10:21:49.928136 24354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } }
I20260619 10:21:49.928279 24354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:49.928318 24354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3205b88e3f4a69a5d947bc5bcd5d62, State: Initialized, Role: FOLLOWER
I20260619 10:21:49.928438 24354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } }
I20260619 10:21:49.928519 24354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:49.928539 24354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:49.928580 24354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:49.929263 24354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } }
I20260619 10:21:49.929342 24354 leader_election.cc:302] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e3205b88e3f4a69a5d947bc5bcd5d62; no voters:
I20260619 10:21:49.929479 24354 leader_election.cc:288] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:49.929505 24359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:49.929653 24359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [term 1 LEADER]: Becoming Leader. State: Replica: 2e3205b88e3f4a69a5d947bc5bcd5d62, State: Running, Role: LEADER
I20260619 10:21:49.929725 24354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:49.929730 24359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } }
I20260619 10:21:49.930045 24361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3205b88e3f4a69a5d947bc5bcd5d62. Latest consensus state: current_term: 1 leader_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } } }
I20260619 10:21:49.930045 24360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3205b88e3f4a69a5d947bc5bcd5d62" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42641 } } }
I20260619 10:21:49.930147 24361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:49.930166 24360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:49.930361 24365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:49.931037 24365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:49.932453 24365 catalog_manager.cc:1347] Generated new cluster ID: f3ccec71b81a48c1a3719d47fb59a2d5
I20260619 10:21:49.932511 24365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:49.937345 24365 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:49.937479 24365 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:49.942389 24365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: Generated new TSK 0
I20260619 10:21:49.942538 24365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:21:49.945077 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--builtin_ntp_servers=127.19.65.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:21:50.027094 24378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.027339 24378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.027391 24378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.028578 24378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.028617 24378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.028642 24378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:50.030161 24378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.030413 24378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.030740 24378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.035578 24384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.035619 24383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.035761 24378 server_base.cc:1034] running on GCE node
W20260619 10:21:50.035555 24386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.036130 24378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.036408 24378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.037599 24378 hybrid_clock.cc:648] HybridClock initialized: now 1781864510037568 us; error 43 us; skew 500 ppm
I20260619 10:21:50.039026 24378 webserver.cc:460] Webserver started at http://127.19.65.129:38963/ using document root <none> and password file <none>
I20260619 10:21:50.039274 24378 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.039377 24378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.039515 24378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.040891 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "0987436f9d0a4ed187724288f1d2aaf9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "b12f5558baf8fc150922b86edf0ef5af"
server_key_iv: "20afbb5da4764c1429f47588afcf2a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.041294 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "0987436f9d0a4ed187724288f1d2aaf9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "b12f5558baf8fc150922b86edf0ef5af"
server_key_iv: "20afbb5da4764c1429f47588afcf2a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.043015 24378 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 10:21:50.043999 24392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.044220 24378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:50.044312 24378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "0987436f9d0a4ed187724288f1d2aaf9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "b12f5558baf8fc150922b86edf0ef5af"
server_key_iv: "20afbb5da4764c1429f47588afcf2a1e"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.044385 24378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:50.049266 24378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:50.049571 24378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:50.049713 24378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:50.049950 24378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:50.050293 24378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:50.050341 24378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.050372 24378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:50.050390 24378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.057168 24378 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40955
I20260619 10:21:50.057240 24505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40955 every 8 connection(s)
I20260619 10:21:50.057547 24378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:50.059727 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24378
I20260619 10:21:50.059820 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:50.060016 19718 external_mini_cluster.cc:1428] Setting key 9b057f7290d2d63f23089244f524df85
I20260619 10:21:50.060920 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.058341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:56099 (local address 127.19.65.190:42641)
0619 10:21:50.058405 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.058407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.058647 (+ 240us) server_negotiation.cc:407] Connection header received
0619 10:21:50.059097 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.059098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.059104 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.059187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.060639 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.060640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.060641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.060646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.060664 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.060668 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.060719 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.060829 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.060845 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:21:50.060961 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--builtin_ntp_servers=127.19.65.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 10:21:50.061426 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.058305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42641 (local address 127.19.65.129:56099)
0619 10:21:50.058516 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.058555 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.059010 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.059239 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.059242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.059249 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.060457 (+ 1208us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.060457 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.060495 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.060530 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.060730 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.060733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.060735 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.061161 (+ 426us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.061227 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":121,"threads_started":1}
I20260619 10:21:50.062381 24506 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42641
I20260619 10:21:50.062482 24506 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:50.062631 24506 heartbeater.cc:511] Master 127.19.65.190:42641 requested a full tablet report, sending...
I20260619 10:21:50.063190 24318 ts_manager.cc:194] Registered new tserver with Master: 0987436f9d0a4ed187724288f1d2aaf9 (127.19.65.129:40955)
W20260619 10:21:50.167356 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.167562 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.167598 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.168833 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.168896 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.168938 24509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:50.170235 24509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.170485 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.170747 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.175478 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.175474 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.175699 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.175684 24509 server_base.cc:1034] running on GCE node
I20260619 10:21:50.175936 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.176177 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.177320 24509 hybrid_clock.cc:648] HybridClock initialized: now 1781864510177315 us; error 29 us; skew 500 ppm
I20260619 10:21:50.178481 24509 webserver.cc:460] Webserver started at http://127.19.65.130:33819/ using document root <none> and password file <none>
I20260619 10:21:50.178704 24509 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.178745 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.178887 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.179911 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "65010c93372c4895b88d055c28aac832"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "1421c923af4f7351e1ff36b8004539cf"
server_key_iv: "03e1dd7eaa3fd82a3f37ed2369d9b4b3"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.180243 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "65010c93372c4895b88d055c28aac832"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "1421c923af4f7351e1ff36b8004539cf"
server_key_iv: "03e1dd7eaa3fd82a3f37ed2369d9b4b3"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.181452 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:50.182176 24523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.182361 24509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:50.182440 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "65010c93372c4895b88d055c28aac832"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "1421c923af4f7351e1ff36b8004539cf"
server_key_iv: "03e1dd7eaa3fd82a3f37ed2369d9b4b3"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.182499 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:50.185659 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:50.185858 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:50.185950 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:50.186093 24509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:50.186389 24509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:50.186422 24509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.186441 24509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:50.186455 24509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.191994 24509 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43699
I20260619 10:21:50.192036 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:43699 every 8 connection(s)
I20260619 10:21:50.192302 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:50.194875 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.192917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33395 (local address 127.19.65.190:42641)
0619 10:21:50.192957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.192959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.193185 (+ 226us) server_negotiation.cc:407] Connection header received
0619 10:21:50.193610 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.193611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.193615 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.193687 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.194703 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.194704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.194704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.194708 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.194721 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.194725 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.194770 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.194815 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.194826 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:50.195066 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.192906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42641 (local address 127.19.65.130:33395)
0619 10:21:50.193067 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.193105 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.193538 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.193734 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.193737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.193745 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.194558 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.194558 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.194583 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.194606 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.194768 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.194770 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.194772 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.194819 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.194863 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260619 10:21:50.195565 24637 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42641
I20260619 10:21:50.195641 24637 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:50.195798 24637 heartbeater.cc:511] Master 127.19.65.190:42641 requested a full tablet report, sending...
I20260619 10:21:50.196128 24318 ts_manager.cc:194] Registered new tserver with Master: 65010c93372c4895b88d055c28aac832 (127.19.65.130:43699)
I20260619 10:21:50.196487 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24509
I20260619 10:21:50.196585 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:50.196760 19718 external_mini_cluster.cc:1428] Setting key 3e0be3098565597bcbd51c922a6f13e5
I20260619 10:21:50.197577 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--builtin_ntp_servers=127.19.65.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:21:50.274586 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.274799 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.274829 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.275986 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.276047 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.276091 24640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:50.277393 24640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.277635 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.277920 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.282625 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.282671 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.282685 24648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.282868 24640 server_base.cc:1034] running on GCE node
I20260619 10:21:50.283061 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.283237 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.284397 24640 hybrid_clock.cc:648] HybridClock initialized: now 1781864510284374 us; error 40 us; skew 500 ppm
I20260619 10:21:50.285566 24640 webserver.cc:460] Webserver started at http://127.19.65.131:46355/ using document root <none> and password file <none>
I20260619 10:21:50.285838 24640 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.285915 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.286052 24640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.287050 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "ab25476ed15143ed8dde27d05b384bd9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "139ffd9c8acfcf2b32ebceb902f8ed51"
server_key_iv: "855b06fc91d80045076f6f7c234f0b1c"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.287436 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "ab25476ed15143ed8dde27d05b384bd9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "139ffd9c8acfcf2b32ebceb902f8ed51"
server_key_iv: "855b06fc91d80045076f6f7c234f0b1c"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.288663 24640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:50.289333 24654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.289484 24640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:50.289575 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "ab25476ed15143ed8dde27d05b384bd9"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "139ffd9c8acfcf2b32ebceb902f8ed51"
server_key_iv: "855b06fc91d80045076f6f7c234f0b1c"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.289654 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:50.292970 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:50.293224 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:50.293357 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:50.293565 24640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:50.293855 24640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:50.293910 24640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.293946 24640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:50.293975 24640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.299232 24640 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40689
I20260619 10:21:50.299314 24767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:40689 every 8 connection(s)
I20260619 10:21:50.299587 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:50.301488 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24640
I20260619 10:21:50.301580 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:50.301759 19718 external_mini_cluster.cc:1428] Setting key 39b5d7b6a0e5e50118c1e49328d2c77b
I20260619 10:21:50.302554 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.300227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:35631 (local address 127.19.65.190:42641)
0619 10:21:50.300293 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.300294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.300548 (+ 254us) server_negotiation.cc:407] Connection header received
0619 10:21:50.301011 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.301012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.301019 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.301077 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.302304 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.302305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.302306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.302311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.302325 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.302329 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.302391 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.302467 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.302484 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:21:50.302735 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.300221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42641 (local address 127.19.65.131:35631)
0619 10:21:50.300434 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.300464 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.300902 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.301144 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.301148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.301155 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.302169 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.302170 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.302197 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.302221 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.302394 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.302398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.302401 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.302462 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.302527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260619 10:21:50.303166 24768 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42641
I20260619 10:21:50.303232 24768 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:50.303416 24768 heartbeater.cc:511] Master 127.19.65.190:42641 requested a full tablet report, sending...
I20260619 10:21:50.303742 24318 ts_manager.cc:194] Registered new tserver with Master: ab25476ed15143ed8dde27d05b384bd9 (127.19.65.131:40689)
I20260619 10:21:50.313400 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:21:50.315650 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.314679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.19.65.190:42641)
0619 10:21:50.314735 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.314737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.314852 (+ 115us) server_negotiation.cc:407] Connection header received
0619 10:21:50.314892 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.314893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.314896 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.314957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.315494 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.315495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.315495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.315499 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.315511 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.315514 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.315562 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.315594 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.315604 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:21:50.317065 24318 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:50.318593 24318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:50.322275 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.320617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40955 (local address 127.0.0.1:39246)
0619 10:21:50.320753 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.320799 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.320865 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.321129 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.321134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.321140 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.321842 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.321843 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.321858 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.321875 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.322135 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.322140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.322143 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.322201 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.322214 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260619 10:21:50.322618 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.320814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:40689 (local address 127.0.0.1:60264)
0619 10:21:50.321070 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.321079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.321122 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.321502 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.321503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.321505 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.322055 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.322055 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.322061 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.322074 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.322474 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.322475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.322476 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.322532 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.322546 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":63,"threads_started":1}
I20260619 10:21:50.322633 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.320742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.19.65.129:40955)
0619 10:21:50.320963 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.320969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.320979 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:50.321009 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.321011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.321021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.321127 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.321980 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.321987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.321987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.321992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.322029 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.322061 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.322110 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.322535 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.322552 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260619 10:21:50.322917 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.320909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.19.65.131:40689)
0619 10:21:50.321323 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.321327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.321338 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:50.321363 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.321365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.321375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.321496 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.322172 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.322180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.322181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.322186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.322223 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.322250 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.322308 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.322819 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.322838 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":79,"threads_started":1}
I20260619 10:21:50.323266 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.320965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:43699 (local address 127.0.0.1:49924)
0619 10:21:50.321833 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.321843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.321900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.322057 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.322058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.322061 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.322496 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.322496 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.322503 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.322515 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.323136 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.323137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.323138 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.323195 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.323208 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":54,"threads_started":1}
I20260619 10:21:50.323406 24440 tablet_service.cc:1467] Processing CreateTablet for tablet c5a6400c56334b8db079f370763cedf3 (DEFAULT_TABLE table=test-table [id=b64e87881aa04cf28050d232744141db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:50.323616 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.321102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.19.65.130:43699)
0619 10:21:50.321268 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.321272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.321933 (+ 661us) server_negotiation.cc:407] Connection header received
0619 10:21:50.321962 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.321965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.321974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.322050 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.322603 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.322609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.322610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.322614 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.322648 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.322678 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.323008 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.323481 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.323498 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260619 10:21:50.323606 24702 tablet_service.cc:1467] Processing CreateTablet for tablet c5a6400c56334b8db079f370763cedf3 (DEFAULT_TABLE table=test-table [id=b64e87881aa04cf28050d232744141db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:50.323776 24440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a6400c56334b8db079f370763cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:50.323961 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a6400c56334b8db079f370763cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:50.324354 24571 tablet_service.cc:1467] Processing CreateTablet for tablet c5a6400c56334b8db079f370763cedf3 (DEFAULT_TABLE table=test-table [id=b64e87881aa04cf28050d232744141db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:50.324644 24571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a6400c56334b8db079f370763cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:50.326719 24782 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Bootstrap starting.
I20260619 10:21:50.327543 24783 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: Bootstrap starting.
I20260619 10:21:50.327579 24782 tablet_bootstrap.cc:654] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:50.327838 24784 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Bootstrap starting.
I20260619 10:21:50.327913 24782 log.cc:824] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:50.328274 24783 tablet_bootstrap.cc:654] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:50.328507 24784 tablet_bootstrap.cc:654] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:50.328626 24783 log.cc:824] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:50.328848 24784 log.cc:824] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:50.329015 24782 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: No bootstrap required, opened a new log
I20260619 10:21:50.329100 24782 ts_tablet_manager.cc:1397] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:21:50.329438 24783 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: No bootstrap required, opened a new log
I20260619 10:21:50.329484 24783 ts_tablet_manager.cc:1397] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:50.329658 24784 tablet_bootstrap.cc:492] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: No bootstrap required, opened a new log
I20260619 10:21:50.329751 24784 ts_tablet_manager.cc:1397] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:50.331621 24783 raft_consensus.cc:348] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.331765 24783 raft_consensus.cc:374] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:50.331806 24783 raft_consensus.cc:729] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0987436f9d0a4ed187724288f1d2aaf9, State: Initialized, Role: FOLLOWER
I20260619 10:21:50.331925 24783 consensus_queue.cc:260] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.332197 24783 ts_tablet_manager.cc:1428] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:21:50.332065 24782 raft_consensus.cc:348] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.332295 24782 raft_consensus.cc:374] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:50.332309 24506 heartbeater.cc:503] Master 127.19.65.190:42641 was elected leader, sending a full tablet report...
I20260619 10:21:50.332338 24782 raft_consensus.cc:729] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab25476ed15143ed8dde27d05b384bd9, State: Initialized, Role: FOLLOWER
I20260619 10:21:50.332454 24782 consensus_queue.cc:260] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.332448 24784 raft_consensus.cc:348] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.332608 24784 raft_consensus.cc:374] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:50.332652 24784 raft_consensus.cc:729] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65010c93372c4895b88d055c28aac832, State: Initialized, Role: FOLLOWER
I20260619 10:21:50.332707 24782 ts_tablet_manager.cc:1428] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 10:21:50.332777 24784 consensus_queue.cc:260] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.332839 24768 heartbeater.cc:503] Master 127.19.65.190:42641 was elected leader, sending a full tablet report...
I20260619 10:21:50.333047 24784 ts_tablet_manager.cc:1428] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:50.333065 24637 heartbeater.cc:503] Master 127.19.65.190:42641 was elected leader, sending a full tablet report...
I20260619 10:21:50.384272 24790 raft_consensus.cc:482] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:50.384382 24790 raft_consensus.cc:504] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.384862 24790 leader_election.cc:288] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0987436f9d0a4ed187724288f1d2aaf9 (127.19.65.129:40955), ab25476ed15143ed8dde27d05b384bd9 (127.19.65.131:40689)
I20260619 10:21:50.386271 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.385147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34131 (local address 127.19.65.129:40955)
0619 10:21:50.385213 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.385215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.385269 (+ 54us) server_negotiation.cc:407] Connection header received
0619 10:21:50.385301 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.385302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.385307 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.385359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.386009 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.386010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.386011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.386016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.386032 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.386037 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.386117 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.386182 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.386196 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:21:50.386279 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.385070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40955 (local address 127.19.65.130:34131)
0619 10:21:50.385132 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.385145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.385198 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.385387 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.385388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.385391 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.385899 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.385900 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.385909 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.385923 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.386117 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.386118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.386119 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.386175 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.386187 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 10:21:50.386585 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.385310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33015 (local address 127.19.65.131:40689)
0619 10:21:50.385386 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.385388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.385397 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:50.385470 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.385471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.385476 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.385530 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.386258 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.386260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.386260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.386265 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.386282 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.386287 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.386348 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.386505 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.386519 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 10:21:50.386605 24460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a6400c56334b8db079f370763cedf3" candidate_uuid: "65010c93372c4895b88d055c28aac832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0987436f9d0a4ed187724288f1d2aaf9" is_pre_election: true
I20260619 10:21:50.386775 24460 raft_consensus.cc:2454] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65010c93372c4895b88d055c28aac832 in term 0.
I20260619 10:21:50.386781 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.385180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:40689 (local address 127.19.65.130:33015)
0619 10:21:50.385343 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.385354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.385407 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.385538 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.385538 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.385541 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:50.386137 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:21:50.386138 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:50.386145 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:21:50.386160 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:50.386418 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:50.386419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:50.386420 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:21:50.386688 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 10:21:50.386703 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260619 10:21:50.387002 24524 leader_election.cc:302] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0987436f9d0a4ed187724288f1d2aaf9, 65010c93372c4895b88d055c28aac832; no voters:
I20260619 10:21:50.387145 24790 raft_consensus.cc:2781] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:50.387130 24722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a6400c56334b8db079f370763cedf3" candidate_uuid: "65010c93372c4895b88d055c28aac832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab25476ed15143ed8dde27d05b384bd9" is_pre_election: true
I20260619 10:21:50.387212 24790 raft_consensus.cc:482] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:50.387238 24790 raft_consensus.cc:3037] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:50.387296 24722 raft_consensus.cc:2454] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65010c93372c4895b88d055c28aac832 in term 0.
I20260619 10:21:50.388001 24790 raft_consensus.cc:504] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.388136 24790 leader_election.cc:288] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [CANDIDATE]: Term 1 election: Requested vote from peers 0987436f9d0a4ed187724288f1d2aaf9 (127.19.65.129:40955), ab25476ed15143ed8dde27d05b384bd9 (127.19.65.131:40689)
I20260619 10:21:50.388311 24460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a6400c56334b8db079f370763cedf3" candidate_uuid: "65010c93372c4895b88d055c28aac832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0987436f9d0a4ed187724288f1d2aaf9"
I20260619 10:21:50.388345 24722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a6400c56334b8db079f370763cedf3" candidate_uuid: "65010c93372c4895b88d055c28aac832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab25476ed15143ed8dde27d05b384bd9"
I20260619 10:21:50.388411 24460 raft_consensus.cc:3037] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:50.388479 24722 raft_consensus.cc:3037] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:50.389081 24460 raft_consensus.cc:2454] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65010c93372c4895b88d055c28aac832 in term 1.
I20260619 10:21:50.389235 24524 leader_election.cc:302] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0987436f9d0a4ed187724288f1d2aaf9, 65010c93372c4895b88d055c28aac832; no voters:
I20260619 10:21:50.389277 24722 raft_consensus.cc:2454] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65010c93372c4895b88d055c28aac832 in term 1.
I20260619 10:21:50.389309 24790 raft_consensus.cc:2781] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:50.389454 24790 raft_consensus.cc:686] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 1 LEADER]: Becoming Leader. State: Replica: 65010c93372c4895b88d055c28aac832, State: Running, Role: LEADER
I20260619 10:21:50.389544 24790 consensus_queue.cc:237] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } }
I20260619 10:21:50.390278 24317 catalog_manager.cc:5495] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65010c93372c4895b88d055c28aac832 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "65010c93372c4895b88d055c28aac832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65010c93372c4895b88d055c28aac832" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43699 } health_report { overall_health: HEALTHY } } }
I20260619 10:21:50.395624 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.394730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.19.65.130:43699)
0619 10:21:50.394769 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.394773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.394786 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:21:50.394812 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.394812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.394821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.394873 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.395465 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.395466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.395466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.395470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.395483 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.395487 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.395540 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.395572 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.395582 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:50.397212 24460 raft_consensus.cc:1261] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 1 FOLLOWER]: Refusing update from remote peer 65010c93372c4895b88d055c28aac832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:50.397200 24722 raft_consensus.cc:1261] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 1 FOLLOWER]: Refusing update from remote peer 65010c93372c4895b88d055c28aac832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:50.397557 24790 consensus_queue.cc:1035] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0987436f9d0a4ed187724288f1d2aaf9" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:50.397691 24792 consensus_queue.cc:1035] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab25476ed15143ed8dde27d05b384bd9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:50.400008 24795 mvcc.cc:205] Tried to move back new op lower bound from 7298517034584743936 to 7298517034556186624. Current Snapshot: MvccSnapshot[applied={T|T < 7298517034584743936}]
I20260619 10:21:50.400444 24797 mvcc.cc:205] Tried to move back new op lower bound from 7298517034584743936 to 7298517034556186624. Current Snapshot: MvccSnapshot[applied={T|T < 7298517034584743936}]
I20260619 10:21:50.402469 24317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:50.402554 24317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:50.403260 24317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e3205b88e3f4a69a5d947bc5bcd5d62: Sending DeleteTablet for 3 replicas of tablet c5a6400c56334b8db079f370763cedf3
I20260619 10:21:50.403635 24440 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a6400c56334b8db079f370763cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:50 UTC) from {username='slave'} at 127.0.0.1:39246
I20260619 10:21:50.403633 24702 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a6400c56334b8db079f370763cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:50 UTC) from {username='slave'} at 127.0.0.1:60264
I20260619 10:21:50.403851 24805 tablet_replica.cc:330] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9: stopping tablet replica
I20260619 10:21:50.403951 24805 raft_consensus.cc:2229] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:50.403952 24571 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a6400c56334b8db079f370763cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:50 UTC) from {username='slave'} at 127.0.0.1:49924
I20260619 10:21:50.404066 24805 raft_consensus.cc:2258] T c5a6400c56334b8db079f370763cedf3 P 0987436f9d0a4ed187724288f1d2aaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:50.404129 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24378
I20260619 10:21:50.403977 24806 tablet_replica.cc:330] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: stopping tablet replica
I20260619 10:21:50.404279 24806 raft_consensus.cc:2229] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:50.404392 24806 raft_consensus.cc:2258] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:50.404534 24807 tablet_replica.cc:330] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: stopping tablet replica
I20260619 10:21:50.404738 24807 raft_consensus.cc:2229] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:50.404834 24807 raft_consensus.cc:2258] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:50.405200 24807 ts_tablet_manager.cc:1905] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:50.406273 24806 ts_tablet_manager.cc:1905] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:50.406771 24807 ts_tablet_manager.cc:1918] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:50.406852 24807 log.cc:1196] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/c5a6400c56334b8db079f370763cedf3
I20260619 10:21:50.407163 24807 ts_tablet_manager.cc:1939] T c5a6400c56334b8db079f370763cedf3 P 65010c93372c4895b88d055c28aac832: Deleting consensus metadata
I20260619 10:21:50.408066 24806 ts_tablet_manager.cc:1918] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:50.408174 24806 log.cc:1196] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/c5a6400c56334b8db079f370763cedf3
I20260619 10:21:50.408191 24302 catalog_manager.cc:4841] TS 65010c93372c4895b88d055c28aac832 (127.19.65.130:43699): tablet c5a6400c56334b8db079f370763cedf3 (table test-table [id=b64e87881aa04cf28050d232744141db]) successfully deleted
I20260619 10:21:50.408422 24806 ts_tablet_manager.cc:1939] T c5a6400c56334b8db079f370763cedf3 P ab25476ed15143ed8dde27d05b384bd9: Deleting consensus metadata
I20260619 10:21:50.408851 24302 catalog_manager.cc:4841] TS ab25476ed15143ed8dde27d05b384bd9 (127.19.65.131:40689): tablet c5a6400c56334b8db079f370763cedf3 (table test-table [id=b64e87881aa04cf28050d232744141db]) successfully deleted
W20260619 10:21:50.411970 24302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40955 (error 108)
I20260619 10:21:50.412196 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24509
I20260619 10:21:50.414077 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.413958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40955 (local address 127.0.0.1:39248)
0619 10:21:50.414012 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.414053 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 10:21:50.414198 24302 catalog_manager.cc:4584] TS 0987436f9d0a4ed187724288f1d2aaf9 (127.19.65.129:40955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5a6400c56334b8db079f370763cedf3: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40955: connect: Connection refused (error 111)
I20260619 10:21:50.417330 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24640
I20260619 10:21:50.421859 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24287
2026-06-19T10:21:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (626 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 10:21:50.441183 19718 mini_oidc.cc:152] Starting JWKS server
I20260619 10:21:50.442178 19718 webserver.cc:460] Webserver started at https://127.0.0.1:46671/ using document root <none> and password file <none>
I20260619 10:21:50.442288 19718 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:21:50.442457 19718 webserver.cc:460] Webserver started at http://127.0.0.1:35433/ using document root <none> and password file <none>
2026-06-19T10:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:50Z Disabled control of system clock
I20260619 10:21:50.449688 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45321
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45321
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/testchainca.cert with env {}
W20260619 10:21:50.528483 24814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.528689 24814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.528713 24814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.529932 24814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:50.529976 24814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.529991 24814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.530059 24814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:50.530088 24814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:21:50.530117 24814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:21:50.530139 24814 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46671/jwks.json
I20260619 10:21:50.531726 24814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.531986 24814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.532222 24814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.537040 24819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.537067 24820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.537063 24822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.537187 24814 server_base.cc:1034] running on GCE node
I20260619 10:21:50.537499 24814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.537809 24814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.538970 24814 hybrid_clock.cc:648] HybridClock initialized: now 1781864510538958 us; error 42 us; skew 500 ppm
I20260619 10:21:50.540216 24814 webserver.cc:460] Webserver started at http://127.19.65.190:37717/ using document root <none> and password file <none>
I20260619 10:21:50.540490 24814 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.540557 24814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.540692 24814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.541726 24814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "6108f4d684b54d8aac740ce286ec7be0"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "64cc135170b953cc4e44616af22d6933"
server_key_iv: "90bc566afb1ddcbeb7243e1b21043428"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.542054 24814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "6108f4d684b54d8aac740ce286ec7be0"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "64cc135170b953cc4e44616af22d6933"
server_key_iv: "90bc566afb1ddcbeb7243e1b21043428"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.543273 24814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:21:50.544081 24828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.544260 24814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:50.544360 24814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "6108f4d684b54d8aac740ce286ec7be0"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "64cc135170b953cc4e44616af22d6933"
server_key_iv: "90bc566afb1ddcbeb7243e1b21043428"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.544441 24814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:50.559979 24814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:50.560261 24814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:50.560402 24814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:50.564196 24814 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45321
I20260619 10:21:50.564227 24880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:45321 every 8 connection(s)
I20260619 10:21:50.564540 24814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:50.565423 24881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:50.567523 24881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0: Bootstrap starting.
I20260619 10:21:50.568025 24881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:50.568246 24881 log.cc:824] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:50.568825 24881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0: No bootstrap required, opened a new log
I20260619 10:21:50.570286 24881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } }
I20260619 10:21:50.570410 24881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:50.570430 24881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6108f4d684b54d8aac740ce286ec7be0, State: Initialized, Role: FOLLOWER
I20260619 10:21:50.570557 24881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } }
I20260619 10:21:50.570645 24881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:50.570667 24881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:50.570698 24881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:50.571121 24881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } }
I20260619 10:21:50.571195 24881 leader_election.cc:302] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6108f4d684b54d8aac740ce286ec7be0; no voters:
I20260619 10:21:50.571355 24881 leader_election.cc:288] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:50.571408 24884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:50.571573 24884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [term 1 LEADER]: Becoming Leader. State: Replica: 6108f4d684b54d8aac740ce286ec7be0, State: Running, Role: LEADER
I20260619 10:21:50.571632 24881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:50.571661 24884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } }
I20260619 10:21:50.572049 24886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6108f4d684b54d8aac740ce286ec7be0. Latest consensus state: current_term: 1 leader_uuid: "6108f4d684b54d8aac740ce286ec7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } } }
I20260619 10:21:50.572130 24886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:50.572433 24893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:50.572113 24885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6108f4d684b54d8aac740ce286ec7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6108f4d684b54d8aac740ce286ec7be0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45321 } } }
I20260619 10:21:50.572645 24885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:50.572911 24893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:50.573719 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24814
I20260619 10:21:50.573825 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:50.573994 19718 external_mini_cluster.cc:1428] Setting key 4ee6397b5a9379e6646e4b40d8074319
I20260619 10:21:50.574385 24893 catalog_manager.cc:1347] Generated new cluster ID: 9c8517f1ad754d6a9170a4c1832bdcad
I20260619 10:21:50.574438 24893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:50.579690 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.574566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.19.65.190:45321)
0619 10:21:50.575489 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.575495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.575517 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:21:50.575597 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.575601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.575649 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.575776 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.576011 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.576847 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.577691 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.578097 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.578222 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.578910 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.578914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.578914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.579069 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.579098 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.579108 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.579181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.579251 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.579429 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":70,"threads_started":1}
I20260619 10:21:50.580494 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:21:50.601732 24893 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:50.602433 24893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:50.606292 24893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6108f4d684b54d8aac740ce286ec7be0: Generated new TSK 0
I20260619 10:21:50.606464 24893 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 10:21:50.639653 24911 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 10:21:50.639780 24911 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 10:21:50.640066 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.617066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.19.65.190:45321)
0619 10:21:50.617128 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.617130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.617298 (+ 168us) server_negotiation.cc:407] Connection header received
0619 10:21:50.617401 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.617401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.617420 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.617474 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:21:50.617634 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.618113 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.619210 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.619424 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.619473 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.619727 (+ 254us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:21:50.639673 (+ 19946us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 10:21:50.639874 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 10:21:50.640129 24904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 10:21:50.640269 24910 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:50.617017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45321 (local address 127.0.0.1:32778)
0619 10:21:50.617226 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.617237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.617305 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.617500 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.617501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.617503 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:21:50.617571 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:50.617573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.618191 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.618192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:50.619133 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:50.619136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.619501 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.619502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:50.619596 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.619620 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:21:50.639833 (+ 20213us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 10:21:50.639952 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:45321: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.65.190:45321: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 10:21:50.640717 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24814
2026-06-19T10:21:50Z chronyd exiting
I20260619 10:21:50.676481 19718 test_util.cc:175] -----------------------------------------------
I20260619 10:21:50.676550 19718 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864489541676-19718-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 10:21:50.677282 19718 mini_oidc.cc:152] Starting JWKS server
I20260619 10:21:50.677930 19718 webserver.cc:460] Webserver started at https://127.0.0.1:34905/ using document root <none> and password file <none>
I20260619 10:21:50.677968 19718 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:21:50.678172 19718 webserver.cc:460] Webserver started at http://127.0.0.1:35867/ using document root <none> and password file <none>
2026-06-19T10:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:50Z Disabled control of system clock
I20260619 10:21:50.685524 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36845
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/testchainca.cert with env {}
W20260619 10:21:50.763971 24917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.764220 24917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.764261 24917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.765419 24917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:50.765487 24917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.765513 24917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.765547 24917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:50.765581 24917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:21:50.765610 24917 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:21:50.765643 24917 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34905/jwks.json
I20260619 10:21:50.767171 24917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.767482 24917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.767702 24917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.772540 24922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.772568 24925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.772548 24923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.772578 24917 server_base.cc:1034] running on GCE node
I20260619 10:21:50.773006 24917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.773347 24917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.774506 24917 hybrid_clock.cc:648] HybridClock initialized: now 1781864510774500 us; error 35 us; skew 500 ppm
I20260619 10:21:50.775758 24917 webserver.cc:460] Webserver started at http://127.19.65.190:35487/ using document root <none> and password file <none>
I20260619 10:21:50.775995 24917 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.776063 24917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.776190 24917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.777179 24917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "2cf56549ccd4a453b7ad125bbd11ff8e"
server_key_iv: "2cd7b1447178c22cbf3445ea07cd6e63"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.777529 24917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "2cf56549ccd4a453b7ad125bbd11ff8e"
server_key_iv: "2cd7b1447178c22cbf3445ea07cd6e63"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.778741 24917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:50.779476 24931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.779639 24917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:50.779740 24917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "2cf56549ccd4a453b7ad125bbd11ff8e"
server_key_iv: "2cd7b1447178c22cbf3445ea07cd6e63"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.779825 24917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:50.787849 24917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:50.788128 24917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:50.788290 24917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:50.792488 24917 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36845
I20260619 10:21:50.792536 24983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36845 every 8 connection(s)
I20260619 10:21:50.792903 24917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:50.793541 24984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:50.796180 24984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: Bootstrap starting.
I20260619 10:21:50.796684 24984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:50.796901 24984 log.cc:824] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:50.797513 24984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: No bootstrap required, opened a new log
I20260619 10:21:50.798964 24984 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } }
I20260619 10:21:50.799091 24984 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:50.799113 24984 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f9c785c08648eaa2c0d0bfe69a5a34, State: Initialized, Role: FOLLOWER
I20260619 10:21:50.799248 24984 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } }
I20260619 10:21:50.799378 24984 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:50.799417 24984 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:50.799468 24984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:50.799576 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 24917
I20260619 10:21:50.799662 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:50.799865 19718 external_mini_cluster.cc:1428] Setting key 06df4f63e6fe8e799d873871973bd5a4
I20260619 10:21:50.800140 24984 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } }
I20260619 10:21:50.800257 24984 leader_election.cc:302] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f9c785c08648eaa2c0d0bfe69a5a34; no voters:
I20260619 10:21:50.800429 24984 leader_election.cc:288] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:50.800515 24988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:50.800670 24984 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:50.800721 24988 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [term 1 LEADER]: Becoming Leader. State: Replica: b1f9c785c08648eaa2c0d0bfe69a5a34, State: Running, Role: LEADER
I20260619 10:21:50.800801 24988 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } }
I20260619 10:21:50.801120 24990 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } } }
I20260619 10:21:50.801210 24990 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:50.801419 24991 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f9c785c08648eaa2c0d0bfe69a5a34. Latest consensus state: current_term: 1 leader_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9c785c08648eaa2c0d0bfe69a5a34" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36845 } } }
I20260619 10:21:50.801540 24991 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34 [sys.catalog]: This master's current role is: LEADER
W20260619 10:21:50.802503 25005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:50.802557 25005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:50.802629 24998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:50.802805 24998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:50.805027 24998 catalog_manager.cc:1347] Generated new cluster ID: 0e02e411b99f4fe7bc38d7970433a141
I20260619 10:21:50.805087 24998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:50.805840 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.800396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.19.65.190:36845)
0619 10:21:50.800652 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.800662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.800700 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:21:50.800861 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.800868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.800936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.801321 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:50.801417 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.802906 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.803834 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.804434 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.804561 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.805193 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:50.805196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:50.805196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:50.805378 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:50.805409 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:50.805419 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:50.805492 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:50.805564 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:21:50.805640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260619 10:21:50.806703 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:21:50.820667 24998 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:50.821211 24998 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:50.827656 24998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f9c785c08648eaa2c0d0bfe69a5a34: Generated new TSK 0
I20260619 10:21:50.827853 24998 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 10:21:50.860522 25014 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 10:21:50.860633 25014 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 10:21:50.860886 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:50.837545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.19.65.190:36845)
0619 10:21:50.837581 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:50.837584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:50.837593 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:50.837745 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:50.837746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:50.837760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:50.837860 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:21:50.838056 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.838792 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.839968 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.840370 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.840437 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.841271 (+ 834us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:21:50.860665 (+ 19394us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 10:21:50.860812 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:21:50.860946 24989 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 10:21:50.860967 25013 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:50.837348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36845 (local address 127.0.0.1:35092)
0619 10:21:50.837516 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:50.837530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:50.837623 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:50.837874 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:50.837875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:50.837876 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:21:50.837932 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:50.837934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.838904 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.838906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:50.839892 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:50.839894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:50.841083 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:50.841084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:50.841146 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:50.841152 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:21:50.860763 (+ 19611us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 10:21:50.860865 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:36845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 10:21:50.861244 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 24917
2026-06-19T10:21:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 10:21:50.898329 19718 mini_oidc.cc:152] Starting JWKS server
I20260619 10:21:50.898952 19718 webserver.cc:460] Webserver started at https://127.0.0.1:46385/ using document root <none> and password file <none>
I20260619 10:21:50.898988 19718 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:21:50.899129 19718 webserver.cc:460] Webserver started at http://127.0.0.1:38209/ using document root <none> and password file <none>
2026-06-19T10:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:50Z Disabled control of system clock
I20260619 10:21:50.906559 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46837
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46837
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 10:21:50.984777 25020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:50.984954 25020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:50.984975 25020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:50.986035 25020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:50.986066 25020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:50.986079 25020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:50.986091 25020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:50.986104 25020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:21:50.986115 25020 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:21:50.986126 25020 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46385/jwks.json
I20260619 10:21:50.987404 25020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:50.987574 25020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:50.987775 25020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:50.992336 25026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.992431 25020 server_base.cc:1034] running on GCE node
W20260619 10:21:50.992305 25028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:50.992316 25025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:50.992727 25020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:50.993036 25020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:50.994179 25020 hybrid_clock.cc:648] HybridClock initialized: now 1781864510994156 us; error 36 us; skew 500 ppm
I20260619 10:21:50.995424 25020 webserver.cc:460] Webserver started at http://127.19.65.190:45471/ using document root <none> and password file <none>
I20260619 10:21:50.995654 25020 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:50.995698 25020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:50.995849 25020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:50.996832 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "6d89d7effc634a24897d886172d015eb"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "eb69a350dccba7b532166649d5e58da9"
server_key_iv: "6db934c7988d1960339e706f8cd47554"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.997171 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "6d89d7effc634a24897d886172d015eb"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "eb69a350dccba7b532166649d5e58da9"
server_key_iv: "6db934c7988d1960339e706f8cd47554"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.998466 25020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:50.999214 25034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:50.999424 25020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:50.999495 25020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "6d89d7effc634a24897d886172d015eb"
format_stamp: "Formatted at 2026-06-19 10:21:50 on dist-test-slave-xd2q"
server_key: "eb69a350dccba7b532166649d5e58da9"
server_key_iv: "6db934c7988d1960339e706f8cd47554"
server_key_version: "encryptionkey@0"
I20260619 10:21:50.999581 25020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:51.035833 25020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:51.036137 25020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:51.036273 25020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:51.040107 25020 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46837
I20260619 10:21:51.040191 25086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46837 every 8 connection(s)
I20260619 10:21:51.040504 25020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:51.040751 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25020
I20260619 10:21:51.040856 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:51.041039 19718 external_mini_cluster.cc:1428] Setting key c143897af6e18d9f183c4c63ffcfa783
I20260619 10:21:51.041237 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:51.044183 25087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb: Bootstrap starting.
I20260619 10:21:51.044822 25087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:51.045100 25087 log.cc:824] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:51.045730 25087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb: No bootstrap required, opened a new log
I20260619 10:21:51.047058 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:51.041922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.19.65.190:46837)
0619 10:21:51.042140 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:51.042147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:51.042170 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:21:51.042616 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:51.042623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:51.042680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:51.042788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:51.043027 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.043944 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.044968 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.045385 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.045510 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.046215 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:51.046249 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:51.046249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:51.046458 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:51.046495 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:51.046531 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:51.046756 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:51.046787 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 10:21:51.046863 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260619 10:21:51.047904 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:21:51.048092 25087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } }
I20260619 10:21:51.048264 25087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:51.048308 25087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d89d7effc634a24897d886172d015eb, State: Initialized, Role: FOLLOWER
I20260619 10:21:51.048444 25087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } }
I20260619 10:21:51.048527 25087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:51.048558 25087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:51.048607 25087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:51.049275 25087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } }
I20260619 10:21:51.049401 25087 leader_election.cc:302] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d89d7effc634a24897d886172d015eb; no voters:
I20260619 10:21:51.049556 25087 leader_election.cc:288] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:51.049590 25092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:51.049746 25092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [term 1 LEADER]: Becoming Leader. State: Replica: 6d89d7effc634a24897d886172d015eb, State: Running, Role: LEADER
I20260619 10:21:51.049795 25087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:51.049834 25092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } }
I20260619 10:21:51.050149 25093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d89d7effc634a24897d886172d015eb. Latest consensus state: current_term: 1 leader_uuid: "6d89d7effc634a24897d886172d015eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } } }
I20260619 10:21:51.050253 25093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:51.050149 25094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d89d7effc634a24897d886172d015eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d89d7effc634a24897d886172d015eb" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46837 } } }
I20260619 10:21:51.050382 25094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:51.050453 25098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:51.050946 25098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:51.052515 25098 catalog_manager.cc:1347] Generated new cluster ID: 3982c6e739c94c8fab7b3d738639570b
I20260619 10:21:51.052564 25098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:51.063783 25098 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:51.064260 25098 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:51.068218 25098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d89d7effc634a24897d886172d015eb: Generated new TSK 0
I20260619 10:21:51.068396 25098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:21:51.083356 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:51.079903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.19.65.190:46837)
0619 10:21:51.079960 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:51.079962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:51.080053 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:21:51.080111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:51.080112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:51.080123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:51.080185 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:21:51.080298 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.081079 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.081806 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.082100 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.082165 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.082303 (+ 138us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:21:51.083148 (+ 845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 10:21:51.083261 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 10:21:51.083402 25090 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 10:21:51.085182 25116 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:51.079808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46837 (local address 127.0.0.1:45240)
0619 10:21:51.079981 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:51.079990 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:51.080050 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:51.080195 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:51.080196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:51.080197 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:21:51.080244 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:51.080246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.081150 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.081150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:51.081724 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:51.081726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.082180 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.082180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:51.082218 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.082222 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:21:51.083242 (+ 1020us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 10:21:51.085081 (+ 1839us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:46837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1654,"thread_start_us":115,"threads_started":1}
I20260619 10:21:51.085743 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25020
E20260619 10:21:51.095589 25117 webserver.cc:533] Webserver: sslize failed:
E20260619 10:21:51.095639 25117 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T10:21:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T10:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:51Z Disabled control of system clock
I20260619 10:21:51.130329 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44007
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44007
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/testchainca.cert with env {}
W20260619 10:21:51.209491 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:51.209702 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:51.209738 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:51.210793 25121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:51.210829 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:51.210842 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:51.210855 25121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:51.210870 25121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:21:51.210882 25121 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:21:51.210902 25121 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 10:21:51.212288 25121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:51.212472 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:51.212625 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:51.217120 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:51.217109 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:51.217149 25121 server_base.cc:1034] running on GCE node
W20260619 10:21:51.217123 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:51.217559 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:51.217880 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:51.219040 25121 hybrid_clock.cc:648] HybridClock initialized: now 1781864511219034 us; error 35 us; skew 500 ppm
I20260619 10:21:51.220395 25121 webserver.cc:460] Webserver started at http://127.19.65.190:45685/ using document root <none> and password file <none>
I20260619 10:21:51.220618 25121 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:51.220685 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:51.220816 25121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:51.221813 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0"
format_stamp: "Formatted at 2026-06-19 10:21:51 on dist-test-slave-xd2q"
server_key: "62db477da07553d6d7af5157e4ee202e"
server_key_iv: "0454a69244d32adb2c80e7a9622c5806"
server_key_version: "encryptionkey@0"
I20260619 10:21:51.222157 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0"
format_stamp: "Formatted at 2026-06-19 10:21:51 on dist-test-slave-xd2q"
server_key: "62db477da07553d6d7af5157e4ee202e"
server_key_iv: "0454a69244d32adb2c80e7a9622c5806"
server_key_version: "encryptionkey@0"
I20260619 10:21:51.223421 25121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:51.224218 25135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:51.224372 25121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:51.224429 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0"
format_stamp: "Formatted at 2026-06-19 10:21:51 on dist-test-slave-xd2q"
server_key: "62db477da07553d6d7af5157e4ee202e"
server_key_iv: "0454a69244d32adb2c80e7a9622c5806"
server_key_version: "encryptionkey@0"
I20260619 10:21:51.224517 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:51.239739 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:51.240017 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:51.240164 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:51.243616 25121 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44007
I20260619 10:21:51.243677 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:44007 every 8 connection(s)
I20260619 10:21:51.243970 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:51.244272 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25121
I20260619 10:21:51.244371 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:51.244558 19718 external_mini_cluster.cc:1428] Setting key 48f16d578a5f79fcfd857b7dcec40a04
I20260619 10:21:51.245064 25188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:51.247869 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: Bootstrap starting.
I20260619 10:21:51.248481 25188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:51.248816 25188 log.cc:824] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:51.249497 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: No bootstrap required, opened a new log
I20260619 10:21:51.249523 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:51.245000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.19.65.190:44007)
0619 10:21:51.245188 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:51.245196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:51.245228 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:21:51.245599 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:51.245605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:51.245657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:51.245758 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:51.245958 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.246768 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.247809 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.248214 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.248296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.248944 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:51.248948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:51.248949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:21:51.249097 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:51.249125 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:21:51.249134 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:51.249211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:51.249287 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:21:51.249379 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260619 10:21:51.250483 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:21:51.251523 25188 raft_consensus.cc:348] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } }
I20260619 10:21:51.251657 25188 raft_consensus.cc:374] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:51.251681 25188 raft_consensus.cc:729] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62a8ee195fd4ee5869a1f2134a2c0c0, State: Initialized, Role: FOLLOWER
I20260619 10:21:51.251824 25188 consensus_queue.cc:260] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } }
I20260619 10:21:51.251890 25188 raft_consensus.cc:388] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:51.251929 25188 raft_consensus.cc:482] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:51.251974 25188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:51.252456 25188 raft_consensus.cc:504] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } }
I20260619 10:21:51.252558 25188 leader_election.cc:302] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c62a8ee195fd4ee5869a1f2134a2c0c0; no voters:
I20260619 10:21:51.252705 25188 leader_election.cc:288] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:51.252775 25193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:51.252926 25193 raft_consensus.cc:686] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [term 1 LEADER]: Becoming Leader. State: Replica: c62a8ee195fd4ee5869a1f2134a2c0c0, State: Running, Role: LEADER
I20260619 10:21:51.252944 25188 sys_catalog.cc:564] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:51.253001 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } }
I20260619 10:21:51.253384 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c62a8ee195fd4ee5869a1f2134a2c0c0. Latest consensus state: current_term: 1 leader_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } } }
I20260619 10:21:51.253364 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62a8ee195fd4ee5869a1f2134a2c0c0" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44007 } } }
I20260619 10:21:51.253486 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:51.253497 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0 [sys.catalog]: This master's current role is: LEADER
W20260619 10:21:51.254604 25209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:51.254666 25209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:51.254714 25202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:51.254848 25202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:51.259384 25202 catalog_manager.cc:1347] Generated new cluster ID: eef3fa637ad344d6aa519b065f9ea798
I20260619 10:21:51.259436 25202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:51.267982 25202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:51.268446 25202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:51.277284 25202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c62a8ee195fd4ee5869a1f2134a2c0c0: Generated new TSK 0
I20260619 10:21:51.277449 25202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:21:51.840003 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:51.283807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.19.65.190:44007)
0619 10:21:51.283853 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:51.283855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:51.283961 (+ 106us) server_negotiation.cc:407] Connection header received
0619 10:21:51.284047 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:51.284048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:51.284059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:51.284146 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:21:51.284284 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.284776 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.285562 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.285761 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.285852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.286042 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:21:51.839660 (+553618us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 10:21:51.839923 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49650: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:21:51.840075 25217 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:51.283739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44007 (local address 127.0.0.1:49650)
0619 10:21:51.283883 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:51.283893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:51.283978 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:51.284143 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:51.284145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:51.284146 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:21:51.284208 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:51.284210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.284878 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.284880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:51.285476 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:51.285478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:51.285855 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:51.285856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:51.285914 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:51.285919 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:21:51.839857 (+553938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 10:21:51.839975 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.65.190:44007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260619 10:21:51.840086 25191 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49650: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 10:21:51.840387 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25121
2026-06-19T10:21:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (734 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:52 dist-test-slave-xd2q krb5kdc[25229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:52 dist-test-slave-xd2q krb5kdc[25229](info): set up 2 sockets
Jun 19 10:21:52 dist-test-slave-xd2q krb5kdc[25229](info): commencing operation
krb5kdc: starting...
W20260619 10:21:53.039175 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:21:53.149958 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42989
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:53.227739 25242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:53.227917 25242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:53.227938 25242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:53.229214 25242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:53.229254 25242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:53.229269 25242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:53.229281 25242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:53.229327 25242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:53.230675 25242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:53.230913 25242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:53.231117 25242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:53.235694 25248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.235678 25247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.235677 25250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:53.235702 25242 server_base.cc:1034] running on GCE node
I20260619 10:21:53.236068 25242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:53.236372 25242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:53.237534 25242 hybrid_clock.cc:648] HybridClock initialized: now 1781864513237515 us; error 32 us; skew 500 ppm
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.245018 25242 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:21:53.245446 25242 webserver.cc:460] Webserver started at http://127.19.65.190:42029/ using document root <none> and password file <none>
I20260619 10:21:53.245659 25242 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:53.245724 25242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:53.245874 25242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:53.246866 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "c01033980031a215f42d95f7de56b3ec"
server_key_iv: "002eb2a95677f36c4e4a655e736db095"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.247213 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "c01033980031a215f42d95f7de56b3ec"
server_key_iv: "002eb2a95677f36c4e4a655e736db095"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.248497 25242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:53.249254 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.249439 25242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:53.249526 25242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "c01033980031a215f42d95f7de56b3ec"
server_key_iv: "002eb2a95677f36c4e4a655e736db095"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.249598 25242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:53.263826 25242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:53.265934 25242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:53.266036 25242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:53.269703 25242 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42989
I20260619 10:21:53.269733 25309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:42989 every 8 connection(s)
I20260619 10:21:53.270087 25242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:53.270918 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:53.273105 25310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8: Bootstrap starting.
I20260619 10:21:53.273653 25310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:53.273856 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25242
I20260619 10:21:53.273958 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:53.274021 25310 log.cc:824] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:53.274163 19718 external_mini_cluster.cc:1428] Setting key ea3a19b22a1b883fde07bfddf47c99c6
I20260619 10:21:53.274804 25310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8: No bootstrap required, opened a new log
I20260619 10:21:53.276877 25310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } }
I20260619 10:21:53.277022 25310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:53.277086 25310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9a837a84bf4f139b7cfefe20cf2ca8, State: Initialized, Role: FOLLOWER
I20260619 10:21:53.277211 25310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } }
I20260619 10:21:53.277304 25310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:53.277346 25310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:53.277395 25310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:53.277897 25310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } }
I20260619 10:21:53.278010 25310 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9a837a84bf4f139b7cfefe20cf2ca8; no voters:
I20260619 10:21:53.278193 25310 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:53.278240 25315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:53.278386 25315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9a837a84bf4f139b7cfefe20cf2ca8, State: Running, Role: LEADER
I20260619 10:21:53.278548 25315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } }
I20260619 10:21:53.278443 25310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:53.279462 25316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9a837a84bf4f139b7cfefe20cf2ca8. Latest consensus state: current_term: 1 leader_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } } }
I20260619 10:21:53.279551 25316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:53.279676 25317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9a837a84bf4f139b7cfefe20cf2ca8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42989 } } }
I20260619 10:21:53.279742 25317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:53.279798 25321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:53.280325 25321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:53.282179 25321 catalog_manager.cc:1347] Generated new cluster ID: e416678a4c234dc582a5b7cbae346af0
I20260619 10:21:53.282227 25321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:53.285548 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.274633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.19.65.190:42989)
0619 10:21:53.274797 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.274807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.274845 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:21:53.275272 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.275278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.275379 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.275513 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.276107 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.276948 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.277836 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.278400 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.278535 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.281322 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.281326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.281327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.281338 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.284049 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.284434 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.284437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.284437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.284525 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.284750 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.284751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.284751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.284995 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.285161 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.285241 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.285365 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260619 10:21:53.302048 25321 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:53.302583 25321 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:53.310112 25321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9a837a84bf4f139b7cfefe20cf2ca8: Generated new TSK 0
I20260619 10:21:53.310298 25321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:21:53.345594 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:53.446457 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:53.446640 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:53.446662 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:53.447733 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:53.447767 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:53.447791 25336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:53.449134 25336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:53.449306 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:53.449468 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:53.454295 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.454313 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.454293 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:53.454526 25336 server_base.cc:1034] running on GCE node
I20260619 10:21:53.454741 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:53.455003 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:53.456159 25336 hybrid_clock.cc:648] HybridClock initialized: now 1781864513456154 us; error 41 us; skew 500 ppm
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.458459 25336 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:21:53.458884 25336 webserver.cc:460] Webserver started at http://127.19.65.129:41885/ using document root <none> and password file <none>
I20260619 10:21:53.459097 25336 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:53.459162 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:53.459342 25336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:53.460266 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "6798e8f0c9ba4cecaaf01ac04a4d6427"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "5002b599e7bd03a230df8e8d6301bd66"
server_key_iv: "d5b4c628df075c1a9f06b6fd994ff281"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.460608 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "6798e8f0c9ba4cecaaf01ac04a4d6427"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "5002b599e7bd03a230df8e8d6301bd66"
server_key_iv: "d5b4c628df075c1a9f06b6fd994ff281"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.461880 25336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:21:53.462558 25351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.462746 25336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:53.462837 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "6798e8f0c9ba4cecaaf01ac04a4d6427"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "5002b599e7bd03a230df8e8d6301bd66"
server_key_iv: "d5b4c628df075c1a9f06b6fd994ff281"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.462919 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:53.479636 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:53.481760 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:53.481911 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:53.482118 25336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:53.482407 25336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:53.482460 25336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.482494 25336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:53.482522 25336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.488363 25336 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:38269
I20260619 10:21:53.488401 25464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:38269 every 8 connection(s)
I20260619 10:21:53.488690 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:53.490231 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25336
I20260619 10:21:53.490362 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:53.490593 19718 external_mini_cluster.cc:1428] Setting key 7a289fb3cd9729881af5a4a7492b974c
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:21:53.499161 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.489388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:60307 (local address 127.19.65.190:42989)
0619 10:21:53.489454 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.489456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.489640 (+ 184us) server_negotiation.cc:407] Connection header received
0619 10:21:53.490784 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.490786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.490799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.490870 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.491712 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.492188 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.493427 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.493763 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.493836 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.496375 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.496376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.496376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.496382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.497736 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.498196 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.498197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.498197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.498244 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.498522 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.498522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.498522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.498618 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.498665 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.498981 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.499039 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 10:21:53.499401 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.489382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42989 (local address 127.19.65.129:60307)
0619 10:21:53.489527 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:53.489557 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:53.490657 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:53.490970 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:53.490974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:53.491295 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:53.491625 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.491631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.492427 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.492428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.493337 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.493339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.493851 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.493851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.493944 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.494448 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:53.494456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:53.496274 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:53.497819 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.497824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.497827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.498119 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.498308 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.498308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.498309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.498453 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.498683 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:53.498688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:53.498856 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 10:21:53.498964 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 10:21:53.499124 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260619 10:21:53.499969 25465 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42989
I20260619 10:21:53.500140 25465 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:53.500332 25465 heartbeater.cc:511] Master 127.19.65.190:42989 requested a full tablet report, sending...
I20260619 10:21:53.500818 25274 ts_manager.cc:194] Registered new tserver with Master: 6798e8f0c9ba4cecaaf01ac04a4d6427 (127.19.65.129:38269)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
I20260619 10:21:53.501564 25274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:60307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:21:53.515887 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:53.610261 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:53.610472 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:53.610508 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:53.611662 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:53.611728 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:53.611770 25470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:53.613178 25470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:53.613430 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:53.613636 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:53.618494 25476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.618472 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:53.618544 25470 server_base.cc:1034] running on GCE node
W20260619 10:21:53.618485 25475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:53.618953 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:53.619234 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:53.620428 25470 hybrid_clock.cc:648] HybridClock initialized: now 1781864513620406 us; error 39 us; skew 500 ppm
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.622691 25470 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:21:53.623070 25470 webserver.cc:460] Webserver started at http://127.19.65.130:40753/ using document root <none> and password file <none>
I20260619 10:21:53.623325 25470 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:53.623399 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:53.623546 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:53.624511 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "ebd12759a29c485fbd38d8e0eb513ac0"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "87ad85dfc699de05508f80bcaaeffbae"
server_key_iv: "c0d8d83afb35ed06830e3e7cd35ad4c6"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.624881 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "ebd12759a29c485fbd38d8e0eb513ac0"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "87ad85dfc699de05508f80bcaaeffbae"
server_key_iv: "c0d8d83afb35ed06830e3e7cd35ad4c6"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.626168 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:53.626879 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.627035 25470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:53.627130 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "ebd12759a29c485fbd38d8e0eb513ac0"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "87ad85dfc699de05508f80bcaaeffbae"
server_key_iv: "c0d8d83afb35ed06830e3e7cd35ad4c6"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.627228 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:53.642190 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:53.644119 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:53.644276 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:53.644506 25470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:53.644786 25470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:53.644837 25470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.644871 25470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:53.644898 25470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.650626 25470 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:44689
I20260619 10:21:53.650656 25598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:44689 every 8 connection(s)
I20260619 10:21:53.650972 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:53.660075 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.651699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:58003 (local address 127.19.65.190:42989)
0619 10:21:53.651792 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.651795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.651999 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:21:53.652953 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.652954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.652966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.653016 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.653598 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.654080 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.655108 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.655396 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.655481 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.657527 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.657529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.657529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.657535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.658695 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.659088 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.659089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.659089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.659133 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.659437 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.659438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.659438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.659535 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.659586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.659859 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.659973 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:21:53.660059 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25470
I20260619 10:21:53.660171 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:53.660228 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.651668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42989 (local address 127.19.65.130:58003)
0619 10:21:53.651871 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:53.651905 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:53.652847 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:53.653093 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:53.653096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:53.653331 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:53.653498 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.653502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.654198 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.654199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.655001 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.655003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.655490 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.655491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.655611 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.655997 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:53.656005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:53.657444 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:53.658787 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.658793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.658795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.659016 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.659214 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.659215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.659215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.659378 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.659597 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:53.659602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:53.659739 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 10:21:53.659838 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 10:21:53.660002 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260619 10:21:53.660357 19718 external_mini_cluster.cc:1428] Setting key ad87aff5ecb3f42f7aa5aa9680c5d184
I20260619 10:21:53.660866 25599 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42989
I20260619 10:21:53.660951 25599 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:53.661144 25599 heartbeater.cc:511] Master 127.19.65.190:42989 requested a full tablet report, sending...
I20260619 10:21:53.661566 25273 ts_manager.cc:194] Registered new tserver with Master: ebd12759a29c485fbd38d8e0eb513ac0 (127.19.65.130:44689)
I20260619 10:21:53.662061 25273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:58003
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:21:53.685500 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:53.763083 25604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:53.763268 25604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:53.763331 25604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:53.764456 25604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:53.764493 25604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:53.764539 25604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:53.766383 25604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:53.766609 25604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:53.766834 25604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:53.771646 25609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.771647 25610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:53.771647 25612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:53.771752 25604 server_base.cc:1034] running on GCE node
I20260619 10:21:53.772122 25604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:53.772377 25604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:53.773550 25604 hybrid_clock.cc:648] HybridClock initialized: now 1781864513773537 us; error 22 us; skew 500 ppm
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.776170 25604 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:21:53.776633 25604 webserver.cc:460] Webserver started at http://127.19.65.131:40157/ using document root <none> and password file <none>
I20260619 10:21:53.776851 25604 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:53.776919 25604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:53.777072 25604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:53.778053 25604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "b9d73cf7258f4482803cfcbea058f3ab"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "fedc2b66574cdb94243152ebd4ca8251"
server_key_iv: "2b69e9e0fadc37f024efa0852d6c4835"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.778465 25604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "b9d73cf7258f4482803cfcbea058f3ab"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "fedc2b66574cdb94243152ebd4ca8251"
server_key_iv: "2b69e9e0fadc37f024efa0852d6c4835"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.780041 25604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:21:53.780977 25619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.781136 25604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:53.781209 25604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "b9d73cf7258f4482803cfcbea058f3ab"
format_stamp: "Formatted at 2026-06-19 10:21:53 on dist-test-slave-xd2q"
server_key: "fedc2b66574cdb94243152ebd4ca8251"
server_key_iv: "2b69e9e0fadc37f024efa0852d6c4835"
server_key_version: "encryptionkey@0"
I20260619 10:21:53.781288 25604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:53.797274 25604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:53.799486 25604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:53.799587 25604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:53.799789 25604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:53.800045 25604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:53.800074 25604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.800105 25604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:53.800117 25604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:53.805647 25604 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:35331
I20260619 10:21:53.805696 25732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:35331 every 8 connection(s)
I20260619 10:21:53.805968 25604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:53.809965 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25604
I20260619 10:21:53.810081 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:53.810309 19718 external_mini_cluster.cc:1428] Setting key d4f6014c7d66f1be0e1b78c1fee0a87b
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:21:53.815001 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.806627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:51479 (local address 127.19.65.190:42989)
0619 10:21:53.806669 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.806671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.806875 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:21:53.807738 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.807739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.807747 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.807796 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.808339 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.808809 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.809747 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.809917 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.810007 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.812548 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.812549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.812549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.812556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.813698 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.814073 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.814075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.814075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.814129 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.814378 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.814379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.814379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.814492 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.814565 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.814810 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.814881 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:53.815196 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.806623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42989 (local address 127.19.65.131:51479)
0619 10:21:53.806757 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:53.806789 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:53.807656 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:53.807843 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:53.807847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:53.808067 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:53.808267 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.808271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.808882 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.808883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.809677 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:53.809679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.810036 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.810037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:53.810232 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.810847 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:53.810857 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:53.812445 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:53.813787 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.813792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.813794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.814000 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.814208 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:53.814209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:53.814209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:53.814311 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:53.814556 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:53.814561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:53.814689 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 10:21:53.814790 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:21:53.814951 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260619 10:21:53.815614 25733 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42989
I20260619 10:21:53.815687 25733 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:53.815811 25733 heartbeater.cc:511] Master 127.19.65.190:42989 requested a full tablet report, sending...
I20260619 10:21:53.816077 25273 ts_manager.cc:194] Registered new tserver with Master: b9d73cf7258f4482803cfcbea058f3ab (127.19.65.131:35331)
I20260619 10:21:53.816474 25273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:51479
I20260619 10:21:53.822058 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/test-admin.keytab.
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.836047 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 10:21:53.837621 25740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:53.836843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.19.65.129:38269)
0619 10:21:53.837019 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.837023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.837032 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:53.837054 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.837056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.837069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.837131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.837502 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44060: BlockingRecv error: recv got EOF from 127.0.0.1:44060 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
W20260619 10:21:53.837858 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.838707 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.839577 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:53.846773 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.840049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.19.65.129:38269)
0619 10:21:53.840105 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.840107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.840118 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:53.840171 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.840172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.840185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.840241 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.840607 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.841302 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.841969 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.842254 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.842314 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.844350 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.844357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.844357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.844363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.845722 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.846008 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.846010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.846011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.846063 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.846261 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.846262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.846262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.846393 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.846485 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.846555 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.846610 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 10:21:53.848315 25740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:53.847892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.19.65.129:38269)
0619 10:21:53.847927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.847933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.847966 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:21:53.848010 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.848010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.848017 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.848091 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.848284 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44090: BlockingRecv error: recv got EOF from 127.0.0.1:44090 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 10:21:53.848457 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.849334 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.850189 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:53.856421 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.850639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.19.65.129:38269)
0619 10:21:53.850673 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.850675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.850711 (+ 36us) server_negotiation.cc:407] Connection header received
0619 10:21:53.850774 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.850774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.850782 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.850817 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.851122 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.851634 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.852285 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.852458 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.852510 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.854426 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.854427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.854427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.854432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.855641 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.855922 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.855922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.855923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.855966 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.856102 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.856103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.856103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.856171 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.856257 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.856314 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.856367 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:21:53.857726 25740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:53.857319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.19.65.129:38269)
0619 10:21:53.857351 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.857352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.857380 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:21:53.857443 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.857444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.857450 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.857507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.857702 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44106: BlockingRecv error: recv got EOF from 127.0.0.1:44106 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 10:21:53.857875 19718 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864489541676-19718-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.858732 19718 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:53.859549 19718 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:21:53.866236 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:53.860018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.19.65.129:38269)
0619 10:21:53.860052 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:53.860054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:53.860062 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:53.860129 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:53.860130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:53.860139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:53.860193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:53.860630 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.861064 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.861741 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:53.861917 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:53.861999 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:53.863958 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:53.863959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:53.863959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:53.863966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:53.865140 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.865475 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.865476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.865477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.865535 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:53.865744 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:53.865745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:53.865745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:53.865840 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:53.865948 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:53.866070 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:21:53.866142 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:53.866828 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25336
I20260619 10:21:53.871963 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25470
I20260619 10:21:53.876123 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25604
I20260619 10:21:53.880211 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25242
2026-06-19T10:21:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1871 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25753](info): set up 2 sockets
Jun 19 10:21:53 dist-test-slave-xd2q krb5kdc[25753](info): commencing operation
krb5kdc: starting...
W20260619 10:21:54.907384 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:54 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab.
I20260619 10:21:55.017613 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46391
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab
--principal=oryx/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:55.095878 25766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:55.096081 25766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:55.096118 25766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:55.097240 25766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:55.097303 25766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:55.097327 25766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:55.097359 25766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:55.097404 25766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:55.098860 25766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab
--principal=oryx/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:55.099109 25766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:55.099362 25766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.104136 25771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.104038 25772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.104097 25766 server_base.cc:1034] running on GCE node
W20260619 10:21:55.104030 25774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.104568 25766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.104851 25766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.106035 25766 hybrid_clock.cc:648] HybridClock initialized: now 1781864515106023 us; error 42 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.108469 25766 init.cc:377] Logged in from keytab as oryx/127.19.65.190@KRBTEST.COM (short username oryx)
I20260619 10:21:55.108892 25766 webserver.cc:460] Webserver started at http://127.19.65.190:38391/ using document root <none> and password file <none>
I20260619 10:21:55.109100 25766 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.109164 25766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.109315 25766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:55.110248 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "f7acac245ba8476787c80a6bf7924059"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "6af6a7573c1322921f1505a422cfa0c8"
server_key_iv: "4846d7e2ea4e68ebd5f3ec2773cbbff9"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.110594 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "f7acac245ba8476787c80a6bf7924059"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "6af6a7573c1322921f1505a422cfa0c8"
server_key_iv: "4846d7e2ea4e68ebd5f3ec2773cbbff9"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.111968 25766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:21:55.112639 25781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.112803 25766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:55.112890 25766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "f7acac245ba8476787c80a6bf7924059"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "6af6a7573c1322921f1505a422cfa0c8"
server_key_iv: "4846d7e2ea4e68ebd5f3ec2773cbbff9"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.112959 25766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.122416 25766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.124512 25766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:55.124697 25766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.128472 25766 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46391
I20260619 10:21:55.128501 25833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46391 every 8 connection(s)
I20260619 10:21:55.128803 25766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:21:55.129699 25834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:55.131546 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25766
I20260619 10:21:55.131662 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:21:55.131865 19718 external_mini_cluster.cc:1428] Setting key 40dc8d7d163908b8353f2f8e08e58ae2
I20260619 10:21:55.131914 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Bootstrap starting.
I20260619 10:21:55.132625 25834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:55.133013 25834 log.cc:824] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:55.133749 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: No bootstrap required, opened a new log
I20260619 10:21:55.135753 25834 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } }
I20260619 10:21:55.135922 25834 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:55.135972 25834 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7acac245ba8476787c80a6bf7924059, State: Initialized, Role: FOLLOWER
I20260619 10:21:55.136091 25834 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } }
I20260619 10:21:55.136186 25834 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:21:55.136229 25834 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:21:55.136273 25834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:55.136966 25834 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } }
I20260619 10:21:55.137073 25834 leader_election.cc:302] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7acac245ba8476787c80a6bf7924059; no voters:
I20260619 10:21:55.137236 25834 leader_election.cc:288] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:21:55.137290 25839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:55.137434 25839 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 LEADER]: Becoming Leader. State: Replica: f7acac245ba8476787c80a6bf7924059, State: Running, Role: LEADER
I20260619 10:21:55.137486 25834 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:55.137532 25839 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } }
I20260619 10:21:55.137797 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7acac245ba8476787c80a6bf7924059. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } }
I20260619 10:21:55.137903 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:55.137806 25841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } }
I20260619 10:21:55.138031 25841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
W20260619 10:21:55.139299 25854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:55.139412 25854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:55.139513 25844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:55.139685 25844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:55.141242 25844 catalog_manager.cc:1347] Generated new cluster ID: a2a65fd65a594d9c973ad0cf77c8103e
I20260619 10:21:55.141291 25844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:55.142638 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.132312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.19.65.190:46391)
0619 10:21:55.132485 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.132492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.132514 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:21:55.133002 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.133009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.133051 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.133190 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.133885 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.134719 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.135839 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.136454 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.136530 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.139258 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.139262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.139263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.139274 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.141105 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.141498 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.141500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.141501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.141596 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.141799 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.141806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.141807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.142071 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.142214 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.142288 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.142431 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260619 10:21:55.170372 25844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:21:55.170889 25844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:55.173789 25844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Generated new TSK 0
I20260619 10:21:55.173982 25844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
I20260619 10:21:55.202457 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:55.280740 25860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:55.280934 25860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:55.280957 25860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:55.282089 25860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:55.282128 25860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:55.282152 25860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:55.283618 25860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:55.283850 25860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:55.284067 25860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.289278 25866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.289347 25865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.289319 25868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.289476 25860 server_base.cc:1034] running on GCE node
I20260619 10:21:55.289700 25860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.289961 25860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.291131 25860 hybrid_clock.cc:648] HybridClock initialized: now 1781864515291113 us; error 51 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.293661 25860 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20260619 10:21:55.294098 25860 webserver.cc:460] Webserver started at http://127.19.65.129:33283/ using document root <none> and password file <none>
I20260619 10:21:55.294277 25860 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.294317 25860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.294402 25860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:55.295475 25860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "089ee58cdbcd41c3a15dd017674b5482"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "8ee13e3c3d10bfb774365b959622c892"
server_key_iv: "6495da2b7cbdf7ce9bdac2ceadbf3ef2"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.295774 25860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "089ee58cdbcd41c3a15dd017674b5482"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "8ee13e3c3d10bfb774365b959622c892"
server_key_iv: "6495da2b7cbdf7ce9bdac2ceadbf3ef2"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.296957 25860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:55.297602 25875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.297792 25860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:55.297880 25860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "089ee58cdbcd41c3a15dd017674b5482"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "8ee13e3c3d10bfb774365b959622c892"
server_key_iv: "6495da2b7cbdf7ce9bdac2ceadbf3ef2"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.297950 25860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.310055 25860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.311808 25860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:55.311959 25860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.312179 25860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:55.312471 25860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:55.312505 25860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.312525 25860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:55.312575 25860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.318408 25860 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:44023
I20260619 10:21:55.318454 25988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:44023 every 8 connection(s)
I20260619 10:21:55.318733 25860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:21:55.327009 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25860
I20260619 10:21:55.327126 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:21:55.327406 19718 external_mini_cluster.cc:1428] Setting key a4cb1416173a959d5e1c71bfbc08e2b8
I20260619 10:21:55.327653 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.319361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:48119 (local address 127.19.65.190:46391)
0619 10:21:55.319423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.319424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.319642 (+ 218us) server_negotiation.cc:407] Connection header received
0619 10:21:55.320145 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.320146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.320158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.320210 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.321091 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.321600 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.322537 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.322745 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.322834 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.324980 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.324982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.324982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.324989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.326167 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.326572 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.326573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.326573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.326632 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.326876 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.326877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.326878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.326985 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.327055 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.327425 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.327564 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:21:55.327839 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.319343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:48119)
0619 10:21:55.319500 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.319531 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.320041 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.320276 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.320280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.320815 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.320989 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.320994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.321688 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.321689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.322446 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.322448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.322841 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.322843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.322959 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.323474 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.323481 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.324879 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.326263 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.326270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.326271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.326495 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.326699 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.326700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.326700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.326799 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.327057 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.327063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.327227 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.327394 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.327561 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260619 10:21:55.328369 25989 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:55.328470 25989 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:55.328631 25989 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:55.329144 25798 ts_manager.cc:194] Registered new tserver with Master: 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
I20260619 10:21:55.329862 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:48119
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
I20260619 10:21:55.351897 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:55.429558 25994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:55.429731 25994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:55.429752 25994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:55.430833 25994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:55.430871 25994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:55.430895 25994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:55.432358 25994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:55.432576 25994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:55.432806 25994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.437729 26002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.437813 25999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.437920 25994 server_base.cc:1034] running on GCE node
W20260619 10:21:55.437799 26000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.438319 25994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.438557 25994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.439742 25994 hybrid_clock.cc:648] HybridClock initialized: now 1781864515439729 us; error 27 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.442159 25994 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20260619 10:21:55.442574 25994 webserver.cc:460] Webserver started at http://127.19.65.130:42995/ using document root <none> and password file <none>
I20260619 10:21:55.442797 25994 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.442840 25994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.442975 25994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:55.444026 25994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "d9389632ff074715883a8444f9286359"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "bf7d54bdf6c3719d457325313ad063ad"
server_key_iv: "0acae512d2f0ae9cc7b0446fc93f0301"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.444376 25994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "d9389632ff074715883a8444f9286359"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "bf7d54bdf6c3719d457325313ad063ad"
server_key_iv: "0acae512d2f0ae9cc7b0446fc93f0301"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.445654 25994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:55.446352 26009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.446651 25994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:55.446762 25994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "d9389632ff074715883a8444f9286359"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "bf7d54bdf6c3719d457325313ad063ad"
server_key_iv: "0acae512d2f0ae9cc7b0446fc93f0301"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.446841 25994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.461918 25994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.463685 25994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:55.463862 25994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.464109 25994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:55.464394 25994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:55.464447 25994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.464488 25994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:55.464515 25994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.470062 25994 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:33029
I20260619 10:21:55.470112 26122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:33029 every 8 connection(s)
I20260619 10:21:55.470429 25994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:55.476074 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 25994
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:21:55.476173 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:21:55.476423 19718 external_mini_cluster.cc:1428] Setting key 95577e97dce95bb76f590f1b10fa4987
I20260619 10:21:55.480688 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.471106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60609 (local address 127.19.65.190:46391)
0619 10:21:55.471168 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.471170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.471400 (+ 230us) server_negotiation.cc:407] Connection header received
0619 10:21:55.471874 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.471875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.471886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.471946 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.472805 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.473297 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.474260 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.474453 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.474508 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.476683 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.476684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.476684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.476692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.479251 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.479788 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.479789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.479789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.479836 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.480100 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.480101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.480101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.480193 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.480242 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.480575 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.480630 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:21:55.480973 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.471086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:60609)
0619 10:21:55.471245 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.471284 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.471762 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.472006 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.472009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.472549 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.472718 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.472722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.473371 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.473371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.474162 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.474164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.474524 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.474524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.474616 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.475057 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.475065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.476582 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.479397 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.479402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.479404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.479711 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.479898 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.479899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.479899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.480035 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.480260 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.480266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.480423 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.480562 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.480723 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260619 10:21:55.481454 26123 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:55.481551 26123 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:55.481750 26123 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:55.482085 25798 ts_manager.cc:194] Registered new tserver with Master: d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:55.482429 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:60609
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
I20260619 10:21:55.501473 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:55.578687 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:55.578866 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:55.578888 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:55.579946 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:55.579979 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:55.580004 26128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:55.581321 26128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:55.581486 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:55.581642 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.586269 26133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.586319 26134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.586338 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.586434 26128 server_base.cc:1034] running on GCE node
I20260619 10:21:55.586613 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.586841 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.588018 26128 hybrid_clock.cc:648] HybridClock initialized: now 1781864515587996 us; error 45 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.590396 26128 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20260619 10:21:55.590849 26128 webserver.cc:460] Webserver started at http://127.19.65.131:46441/ using document root <none> and password file <none>
I20260619 10:21:55.591073 26128 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.591138 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.591266 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:55.592238 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "dfe9df95a5ef4c83a0384fbe432ec39d"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "4b22ac8e5d0e74c9e0543548693f45d7"
server_key_iv: "e963decf23c9d60047f2444c7b05abfc"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.592584 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "dfe9df95a5ef4c83a0384fbe432ec39d"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "4b22ac8e5d0e74c9e0543548693f45d7"
server_key_iv: "e963decf23c9d60047f2444c7b05abfc"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.593814 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:55.594532 26143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.594717 26128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:55.594807 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "dfe9df95a5ef4c83a0384fbe432ec39d"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "4b22ac8e5d0e74c9e0543548693f45d7"
server_key_iv: "e963decf23c9d60047f2444c7b05abfc"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.594890 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.620651 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.622830 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:55.622979 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.623181 26128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:55.623513 26128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:55.623564 26128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.623602 26128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:55.623627 26128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.629379 26128 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:37843
I20260619 10:21:55.629416 26256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:37843 every 8 connection(s)
I20260619 10:21:55.629737 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:21:55.635756 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26128
I20260619 10:21:55.635864 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:21:55.636046 19718 external_mini_cluster.cc:1428] Setting key 610886a477245ee3ca7e1f6243156ffd
I20260619 10:21:55.638945 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.630398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38679 (local address 127.19.65.190:46391)
0619 10:21:55.630456 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.630459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.630666 (+ 207us) server_negotiation.cc:407] Connection header received
0619 10:21:55.631130 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.631131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.631140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.631190 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.632123 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.632642 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.633562 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.633864 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.633934 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.636124 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.636125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.636125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.636133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.637503 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.637966 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.637968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.637968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.638029 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.638297 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.638298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.638298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.638408 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.638485 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.638785 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.638860 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:21:55.639111 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.630377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:38679)
0619 10:21:55.630524 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.630555 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.631044 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.631241 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.631244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.631849 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.632021 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.632026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.632718 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.632719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.633485 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.633487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.633952 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.633953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.634054 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.634554 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.634561 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.636031 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.637622 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.637628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.637630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.637854 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.638097 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.638098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.638098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.638230 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.638476 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.638482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.638615 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.638722 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.638868 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260619 10:21:55.639576 26257 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:55.639662 26257 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:55.639786 26257 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:55.640125 25798 ts_manager.cc:194] Registered new tserver with Master: dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843)
I20260619 10:21:55.640488 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:38679
I20260619 10:21:55.647749 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.65.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.189.keytab.
Entry for principal oryx/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.189.keytab.
I20260619 10:21:55.670028 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:39195
--webserver_interface=127.19.65.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.189.keytab
--principal=oryx/127.19.65.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46391,127.19.65.189:39195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:55.751513 26262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:55.751732 26262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:55.751770 26262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:55.752912 26262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:55.752977 26262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:55.753001 26262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:55.753029 26262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:55.753073 26262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:55.754592 26262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46391,127.19.65.189:39195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.189.keytab
--principal=oryx/127.19.65.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:39195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.65.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
W20260619 10:21:55.754719 26262 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.65.190:46391,127.19.65.189:39195'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 10:21:55.754878 26262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:55.755087 26262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.759522 26270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.759727 26268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.759809 26267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.759825 26262 server_base.cc:1034] running on GCE node
I20260619 10:21:55.760038 26262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.760272 26262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.761415 26262 hybrid_clock.cc:648] HybridClock initialized: now 1781864515761402 us; error 25 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.763759 26262 init.cc:377] Logged in from keytab as oryx/127.19.65.189@KRBTEST.COM (short username oryx)
I20260619 10:21:55.764230 26262 webserver.cc:460] Webserver started at http://127.19.65.189:46269/ using document root <none> and password file <none>
I20260619 10:21:55.764487 26262 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.764550 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.764679 26262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:55.765658 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/instance:
uuid: "ce218024e9854da0ad769ecd889cb7ea"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "90a1d6cd065f834bffddfa8f9085ee83"
server_key_iv: "0abf30fbda4c432ba4f4b8be7ae2e42b"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.766000 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal/instance:
uuid: "ce218024e9854da0ad769ecd889cb7ea"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "90a1d6cd065f834bffddfa8f9085ee83"
server_key_iv: "0abf30fbda4c432ba4f4b8be7ae2e42b"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.767220 26262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:55.767906 26277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.768064 26262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.768155 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "ce218024e9854da0ad769ecd889cb7ea"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "90a1d6cd065f834bffddfa8f9085ee83"
server_key_iv: "0abf30fbda4c432ba4f4b8be7ae2e42b"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.768246 26262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.794628 26262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.796391 26262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:55.796540 26262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.800606 26262 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:39195
I20260619 10:21:55.800632 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.189:39195 every 8 connection(s)
I20260619 10:21:55.800953 26262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
I20260619 10:21:55.801891 26330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:55.803330 26330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:21:55.804989 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26262
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20260619 10:21:55.812601 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.803984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.19.65.190:46391)
0619 10:21:55.804031 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.804033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.804263 (+ 230us) server_negotiation.cc:407] Connection header received
0619 10:21:55.804676 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.804677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.804686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.804735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.805327 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.806031 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.807128 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.807428 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.807479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.809589 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.809591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.809592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.809601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.811136 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.811639 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.811640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.811641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.811705 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.811951 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.811952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.811953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.812096 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.812168 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.812429 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.812514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:21:55.812738 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.803966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42356)
0619 10:21:55.804136 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.804168 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.804601 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.804782 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.804785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.805006 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.805239 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.805243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.806144 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.806145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.807044 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.807046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.807500 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.807501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.807633 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.808088 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.808098 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.809501 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.811204 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.811208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.811210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.811566 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.811775 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.811776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.811776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.811881 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.812169 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.812175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.812299 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.812394 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.812525 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 10:21:55.813390 26330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:21:55.814023 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.805468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.19.65.189:39195)
0619 10:21:55.805614 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.805618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.805641 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:21:55.805682 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.805684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.805708 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.805786 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.806468 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.807247 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.808282 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.808561 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.808614 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.811122 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.811128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.811129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.811135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.812786 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.813174 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.813177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.813178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.813244 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.813446 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.813447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.813448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.813607 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.813767 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.813829 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.813901 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20260619 10:21:55.819777 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.813658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60626)
0619 10:21:55.813686 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.813694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.813730 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.813928 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.813928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.814051 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.814117 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.814119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.814809 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.814810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.815390 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.815392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.815733 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.815733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.815780 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.816055 (+ 275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.816059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.817080 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.818944 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.818945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.818945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.819089 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.819253 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.819254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.819254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.819353 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.819573 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.819573 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.819604 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.819653 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.819726 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260619 10:21:55.819892 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.813690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.19.65.189:39195)
0619 10:21:55.813820 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.813821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.813827 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:21:55.813843 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.813843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.813854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.813917 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.814198 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.814748 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.815470 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.815663 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.815715 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.817152 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.817153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.817153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.817159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.818875 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.819152 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.819153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.819153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.819199 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.819434 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.819434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.819435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.819519 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.819567 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.819666 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.819770 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
I20260619 10:21:55.820942 26330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Bootstrap starting.
I20260619 10:21:55.821521 26330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:55.821760 26330 log.cc:824] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:55.822471 26330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: No bootstrap required, opened a new log
I20260619 10:21:55.823776 26330 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } }
I20260619 10:21:55.823894 26330 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:55.823956 26330 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce218024e9854da0ad769ecd889cb7ea, State: Initialized, Role: FOLLOWER
I20260619 10:21:55.824095 26330 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } }
I20260619 10:21:55.824352 26337 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } } }
I20260619 10:21:55.824411 26330 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:55.824431 26337 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:21:55.825904 26348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:55.826112 26348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:55.827440 26262 master_runner.cc:428] Detected that this master ce218024e9854da0ad769ecd889cb7ea is joining an existing cluster
I20260619 10:21:55.827503 26262 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.65.189:39195
I20260619 10:21:55.835016 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.828425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.19.65.190:46391)
0619 10:21:55.828476 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.828478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.828637 (+ 159us) server_negotiation.cc:407] Connection header received
0619 10:21:55.828731 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.828732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.828741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.828791 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.829140 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.829849 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.830896 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.831185 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.831253 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.832260 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.832261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.832261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.832267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.833692 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834200 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.834201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.834201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.834254 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834447 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.834448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.834449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.834569 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.834627 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.834821 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.834893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:21:55.835027 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.828469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42372)
0619 10:21:55.828579 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.828589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.828660 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.828794 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.828794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.828992 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.829052 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.829053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.829943 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.829944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.830817 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.830819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.831332 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.831333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.831403 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.831788 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.831792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.832158 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.833788 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.833789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.833789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.834113 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.834334 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.834335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.834401 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.834649 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.834650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.834688 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.834786 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.834883 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260619 10:21:55.835281 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.828598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.19.65.189:39195)
0619 10:21:55.828626 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.828627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.828635 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:55.828662 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.828663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.828670 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.828709 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.829071 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.829511 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.830175 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.830394 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.830473 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.831419 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.831420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.831420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.831427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.834172 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834507 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.834507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.834507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.834546 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834757 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.834758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.834758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.834864 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.834934 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.835105 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.835186 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:21:55.835551 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.828423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60628)
0619 10:21:55.828553 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.828563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.828614 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.828739 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.828740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.828940 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.828992 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.828994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.829585 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.829587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.830092 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.830093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.830490 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.830491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.830547 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.830903 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.830907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.831317 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.834252 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.834253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.834441 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.834611 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.834612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.834612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.834682 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.834955 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.834956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.835004 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.835364 (+ 360us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.835461 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20260619 10:21:55.844167 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.837343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.19.65.189:39195)
0619 10:21:55.837374 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.837376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.837531 (+ 155us) server_negotiation.cc:407] Connection header received
0619 10:21:55.837624 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.837624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.837631 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.837669 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.838139 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.838633 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.839651 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.839810 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.839858 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.841719 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.841720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.841720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.841727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.843008 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.843431 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.843432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.843432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.843481 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.843667 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.843668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.843668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.843757 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.843837 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.844022 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.844091 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:21:55.844182 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.837304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60644)
0619 10:21:55.837446 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.837479 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.837544 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.837684 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.837686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.837989 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.838042 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.838043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.838712 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.838713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.839584 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.839586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.839880 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.839881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.839959 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.840308 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.840313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.841633 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.843119 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.843125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.843127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.843367 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.843543 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.843544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.843544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.843599 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.843834 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.843841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.843887 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.843987 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.844094 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260619 10:21:55.844660 25798 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.65.189:39195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea"
member_type: NON_VOTER
last_known_addr {
host: "127.19.65.189"
port: 39195
}
attrs {
promote: true
}
}
dest_uuid: "f7acac245ba8476787c80a6bf7924059"
cas_config_opid_index: -1
I20260619 10:21:55.844929 25798 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } }
I20260619 10:21:55.845321 26358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ce218024e9854da0ad769ecd889cb7ea (127.19.65.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } }
I20260619 10:21:55.845532 26304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:55.846318 26304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 10:21:55.846349 26358 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.65.189:39195
I20260619 10:21:55.846423 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } }
I20260619 10:21:55.846517 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:55.846658 26358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:55.846727 26262 master.cc:537] Master@127.19.65.189:39195 shutting down...
W20260619 10:21:55.847431 25782 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 10:21:55.849393 26262 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:55.849484 26262 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:55.849546 26262 tablet_replica.cc:330] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: stopping tablet replica
I20260619 10:21:55.851613 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.851505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60652)
0619 10:21:55.851548 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.851595 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:21:55.851747 25782 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 -> Peer ce218024e9854da0ad769ecd889cb7ea (127.19.65.189:39195): Couldn't send request to peer ce218024e9854da0ad769ecd889cb7ea. Status: Network error: Client connection negotiation failed: client connection to 127.19.65.189:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 10:21:55.861966 26262 master.cc:552] Master@127.19.65.189:39195 shutdown complete.
I20260619 10:21:55.862027 26262 master_runner.cc:305] Clearing existing system tablet
I20260619 10:21:55.862142 26262 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.862198 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.862762 26262 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:55.863198 26363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.863346 26262 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.863412 26262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.863456 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "ce218024e9854da0ad769ecd889cb7ea"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "90a1d6cd065f834bffddfa8f9085ee83"
server_key_iv: "0abf30fbda4c432ba4f4b8be7ae2e42b"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.864001 26262 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:55.865315 26262 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 10:21:55.865367 26262 log.cc:1196] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 10:21:55.865557 26262 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Deleting consensus metadata
I20260619 10:21:55.865689 26262 master_runner.cc:315] Copying system tablet from 127.19.65.190:46391
I20260619 10:21:55.866271 26262 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: tablet copy: Beginning tablet copy session from remote peer at address 127.19.65.190:46391
I20260619 10:21:55.872087 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.866530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42376)
0619 10:21:55.866665 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:55.866674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:55.866728 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:55.866865 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:55.866865 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:55.866986 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:55.867033 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.867034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.867979 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.867979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.868489 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:55.868490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.868935 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.868935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:55.868979 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.869413 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:55.869417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:55.869679 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:55.871200 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.871200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.871201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.871415 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.871596 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:55.871597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:55.871597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:55.871643 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:55.871882 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:55.871882 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:55.871913 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 10:21:55.871959 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:21:55.872023 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260619 10:21:55.872130 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.866587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.19.65.190:46391)
0619 10:21:55.866625 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.866627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.866741 (+ 114us) server_negotiation.cc:407] Connection header received
0619 10:21:55.866773 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.866774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.866788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.866863 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.867092 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.867908 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.868564 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.868864 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.868931 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.869757 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.869758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.869759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.869765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.871123 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.871477 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.871478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.871478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.871533 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.871709 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.871710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.871710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.871824 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.871887 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.871983 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.872048 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:55.872372 25818 tablet_copy_service.cc:140] P f7acac245ba8476787c80a6bf7924059: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ce218024e9854da0ad769ecd889cb7ea ({username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:42376)
I20260619 10:21:55.872454 25818 tablet_copy_service.cc:161] P f7acac245ba8476787c80a6bf7924059: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ce218024e9854da0ad769ecd889cb7ea at {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:42376: session id = ce218024e9854da0ad769ecd889cb7ea-00000000000000000000000000000000
I20260619 10:21:55.872977 25818 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 10:21:55.873481 26262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 10:21:55.874197 26262 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: tablet copy: Starting download of 0 data blocks...
I20260619 10:21:55.874305 26262 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: tablet copy: Starting download of 1 WAL segments...
I20260619 10:21:55.875093 26262 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 10:21:55.875854 25818 tablet_copy_service.cc:342] P f7acac245ba8476787c80a6bf7924059: Request end of tablet copy session ce218024e9854da0ad769ecd889cb7ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:42376
I20260619 10:21:55.875910 25818 tablet_copy_service.cc:433] P f7acac245ba8476787c80a6bf7924059: ending tablet copy session ce218024e9854da0ad769ecd889cb7ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ce218024e9854da0ad769ecd889cb7ea
I20260619 10:21:55.878123 26262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:55.879192 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.879321 26371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:55.879406 26370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:55.879489 26262 server_base.cc:1034] running on GCE node
I20260619 10:21:55.879587 26262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:55.879726 26262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:55.880856 26262 hybrid_clock.cc:648] HybridClock initialized: now 1781864515880850 us; error 24 us; skew 500 ppm
Jun 19 10:21:55 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:55.882050 26262 init.cc:377] Logged in from keytab as oryx/127.19.65.189@KRBTEST.COM (short username oryx)
I20260619 10:21:55.882315 26262 webserver.cc:460] Webserver started at http://127.19.65.189:40813/ using document root <none> and password file <none>
I20260619 10:21:55.882440 26262 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:55.882479 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:55.883006 26262 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.883487 26380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.883610 26262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:55.883651 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "ce218024e9854da0ad769ecd889cb7ea"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "90a1d6cd065f834bffddfa8f9085ee83"
server_key_iv: "0abf30fbda4c432ba4f4b8be7ae2e42b"
server_key_version: "encryptionkey@0"
I20260619 10:21:55.883693 26262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:55.912014 26262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:55.913507 26262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:55.916656 26262 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:39195
I20260619 10:21:55.916710 26432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.189:39195 every 8 connection(s)
I20260619 10:21:55.916968 26262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
I20260619 10:21:55.917761 26433 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:21:55.918251 26433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Bootstrap starting.
I20260619 10:21:55.920965 26433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:21:55.921180 26433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Bootstrap complete.
I20260619 10:21:55.921363 26433 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } }
I20260619 10:21:55.921443 26433 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce218024e9854da0ad769ecd889cb7ea, State: Initialized, Role: LEARNER
I20260619 10:21:55.921478 26433 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } }
I20260619 10:21:55.921638 26433 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:55.921635 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } }
I20260619 10:21:55.921787 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: LEARNER
I20260619 10:21:55.923161 26447 catalog_manager.cc:1259] Loaded cluster ID: a2a65fd65a594d9c973ad0cf77c8103e
I20260619 10:21:55.923323 26447 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: loading cluster ID for follower catalog manager: success
I20260619 10:21:55.924392 26447 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: acquiring CA information for follower catalog manager: success
I20260619 10:21:55.924649 26447 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:21:55.972419 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:55.966006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.19.65.189:39195)
0619 10:21:55.966147 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:55.966149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:55.966162 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:21:55.966184 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:55.966184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:55.966194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:55.966270 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:55.966830 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.967568 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.968286 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:55.968473 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:55.968544 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:55.969743 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:55.969744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:55.969745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:55.969752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:55.971248 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.971646 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.971647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.971647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.971712 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:55.971912 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:55.971913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:55.971914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:55.972020 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:55.972122 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:55.972235 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 10:21:55.972313 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.19.65.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.188.keytab.
Entry for principal oryx/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.188.keytab.
I20260619 10:21:55.995898 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:40259
--webserver_interface=127.19.65.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.188.keytab
--principal=oryx/127.19.65.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:56.074738 26451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:56.074975 26451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:56.075009 26451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:56.076182 26451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:56.076244 26451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:56.076267 26451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:56.076344 26451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:56.076375 26451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:56.077796 26451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.188.keytab
--principal=oryx/127.19.65.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.65.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:56.078047 26451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:56.078248 26451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:56.082566 26459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.082693 26451 server_base.cc:1034] running on GCE node
W20260619 10:21:56.082599 26457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.082556 26456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.083009 26451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:56.083339 26451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:56.084528 26451 hybrid_clock.cc:648] HybridClock initialized: now 1781864516084519 us; error 25 us; skew 500 ppm
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:56.087054 26451 init.cc:377] Logged in from keytab as oryx/127.19.65.188@KRBTEST.COM (short username oryx)
I20260619 10:21:56.087520 26451 webserver.cc:460] Webserver started at http://127.19.65.188:39927/ using document root <none> and password file <none>
I20260619 10:21:56.087755 26451 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.087818 26451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.087976 26451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:56.088948 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/instance:
uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
format_stamp: "Formatted at 2026-06-19 10:21:56 on dist-test-slave-xd2q"
server_key: "3c543bda46c7238ba6f0f2b4d0dc9aac"
server_key_iv: "6356b900b37b68eafc8ec48dd3848158"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.089282 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal/instance:
uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
format_stamp: "Formatted at 2026-06-19 10:21:56 on dist-test-slave-xd2q"
server_key: "3c543bda46c7238ba6f0f2b4d0dc9aac"
server_key_iv: "6356b900b37b68eafc8ec48dd3848158"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.090523 26451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:56.091195 26466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.091394 26451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.091490 26451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
format_stamp: "Formatted at 2026-06-19 10:21:56 on dist-test-slave-xd2q"
server_key: "3c543bda46c7238ba6f0f2b4d0dc9aac"
server_key_iv: "6356b900b37b68eafc8ec48dd3848158"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.091570 26451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.113262 26451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:56.115630 26451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:56.115808 26451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:56.119571 26451 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:40259
I20260619 10:21:56.119606 26518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.188:40259 every 8 connection(s)
I20260619 10:21:56.119946 26451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
I20260619 10:21:56.120868 26519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:56.122434 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:21:56.130067 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26451
I20260619 10:21:56.131196 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.123177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.19.65.190:46391)
0619 10:21:56.123242 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.123244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.123333 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:21:56.123754 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.123755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.123780 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.123836 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.124400 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.124862 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.125804 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.126002 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.126054 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.128134 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.128135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.128135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.128141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.129584 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.130056 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.130058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.130058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.130122 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.130469 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.130470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.130470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.130630 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.130716 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.131016 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.131111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:21:56.131433 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.123025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42382)
0619 10:21:56.123175 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.123250 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.123667 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.123902 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.123905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.124119 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.124321 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.124325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.124937 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.124938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.125731 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.125733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.126099 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.126100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.126224 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.126623 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.126630 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.128056 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.129666 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.129672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.129674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.129965 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.130240 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.130241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.130242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.130371 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.130710 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.130714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.130848 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.130978 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.131156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260619 10:21:56.132084 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20260619 10:21:56.138676 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.132386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60676)
0619 10:21:56.132431 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.132440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.132473 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.132617 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.132618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.132745 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.132786 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.132787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.133435 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.133436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.133946 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.133947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.134335 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.134336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.134418 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.134811 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.134816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.136004 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.137612 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.137613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.137613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.137872 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.138065 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.138066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.138066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.138112 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.138448 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.138449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.138480 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.138532 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.138622 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:21:56.138801 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.132464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.19.65.189:39195)
0619 10:21:56.132511 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.132513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.132523 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:56.132546 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.132546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.132559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.132614 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.132860 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.133334 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.134028 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.134257 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.134326 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.136097 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.136098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.136099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.136105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.137545 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.137937 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.137938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.137938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.137999 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.138219 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.138220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.138220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.138342 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.138455 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.138611 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.138693 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:21:56.139142 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:21:56.140105 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.130653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.19.65.188:40259)
0619 10:21:56.130800 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.130804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.130818 (+ 14us) server_negotiation.cc:407] Connection header received
0619 10:21:56.130834 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.130835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.130845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.130926 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.131722 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.132617 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.133333 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.133641 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.133721 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.136820 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.136827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.136827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.136835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.138842 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.139200 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.139202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.139202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.139281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.139517 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.139518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.139518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.139689 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.139819 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.139929 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.140003 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20260619 10:21:56.145437 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.139377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.0.0.1:49494)
0619 10:21:56.139407 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.139420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.139448 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.139660 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.139660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.139814 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.139846 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.139848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.140630 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.140631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.141096 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.141098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.141419 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.141419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.141452 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.141791 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.141795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.142791 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.144663 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.144663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.144664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.144814 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.144996 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.144997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.144997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.145042 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.145234 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.145234 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.145264 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.145321 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.145385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 10:21:56.145550 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.139439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.19.65.188:40259)
0619 10:21:56.139560 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.139562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.139567 (+ 5us) server_negotiation.cc:407] Connection header received
0619 10:21:56.139581 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.139581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.139589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.139648 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.139913 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.140550 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.141168 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.141359 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.141407 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.142856 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.142857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.142857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.142862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.144508 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.144878 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.144879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.144879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.144922 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.145091 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.145091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.145092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.145172 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.145225 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.145334 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.145427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260619 10:21:56.146610 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Bootstrap starting.
I20260619 10:21:56.147171 26519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:56.147488 26519 log.cc:824] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:56.148147 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: No bootstrap required, opened a new log
I20260619 10:21:56.149574 26519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } }
I20260619 10:21:56.149698 26519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:56.149739 26519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efcfa1b1e4f4e83bc2aa6d0051e0c19, State: Initialized, Role: FOLLOWER
I20260619 10:21:56.149860 26519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } }
I20260619 10:21:56.150141 26525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } } }
I20260619 10:21:56.150197 26519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:56.150223 26525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:21:56.151822 26536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:21:56.151885 26536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:21:56.153967 26451 master_runner.cc:428] Detected that this master 6efcfa1b1e4f4e83bc2aa6d0051e0c19 is joining an existing cluster
I20260619 10:21:56.154014 26451 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.65.188:40259
I20260619 10:21:56.161682 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.154949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.0.0.1:49500)
0619 10:21:56.155226 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.155235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.155313 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.155487 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.155487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.155745 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.155809 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.155810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156764 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.156764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.157668 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.157670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158211 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.158263 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.158892 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.158896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.159176 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.160650 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.160651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.160651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.160806 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.161022 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.161022 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.161023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.161097 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.161346 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.161347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.161381 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.161447 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.161552 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260619 10:21:56.161705 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.155119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.19.65.188:40259)
0619 10:21:56.155148 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.155150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.155276 (+ 126us) server_negotiation.cc:407] Connection header received
0619 10:21:56.155385 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.155386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.155405 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.155435 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.155889 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156672 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.157754 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158067 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158124 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.159317 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.159318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.159318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.159324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.160557 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.160884 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.160885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.160885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.160942 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.161174 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.161175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.161175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.161282 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.161353 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.161477 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.161601 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":15}
I20260619 10:21:56.183441 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.155155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.19.65.189:39195)
0619 10:21:56.155212 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.155214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.155245 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:21:56.155490 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.155491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.155502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.155532 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.155957 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156673 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.157763 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158075 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158128 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.180417 (+ 22289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.180419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.180419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.180427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.182251 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182620 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.182621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.182621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.182679 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182862 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.182863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.182863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.182996 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.183069 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.183229 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.183353 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:21:56.183482 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.154960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.19.65.190:46391)
0619 10:21:56.155001 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.155003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.155423 (+ 420us) server_negotiation.cc:407] Connection header received
0619 10:21:56.155454 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.155455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.155467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.155525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.155951 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156674 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.157835 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158167 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158205 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.180417 (+ 22212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.180419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.180419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.180428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.182094 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182486 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.182487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.182487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.182541 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182904 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.182905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.182905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.183026 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.183084 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.183282 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.183395 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:56.183965 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.154990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60680)
0619 10:21:56.155186 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.155194 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.155258 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.155595 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.155596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.155820 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.155876 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.155877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156764 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.156764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.157677 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.157678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158205 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.158263 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.179888 (+ 21625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.179892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.180279 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.182338 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.182339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.182547 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.182743 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.182744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.182790 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.183069 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.183069 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.183121 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.183191 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.183842 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":53,"thread_start_us":70,"threads_started":1}
I20260619 10:21:56.183979 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.155057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42394)
0619 10:21:56.155353 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.155362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.155410 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.155570 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.155571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.155802 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.155862 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.155863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.156771 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.156772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.157738 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.157740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.158305 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.158306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.158387 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.179826 (+ 21439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.179831 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.180333 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.182187 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.182188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.182408 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.182762 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.182763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.182763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.182818 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.183112 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.183113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.183152 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.183609 (+ 457us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.183904 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":239,"thread_start_us":56,"threads_started":1}
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864515, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20260619 10:21:56.191339 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25860
I20260619 10:21:56.192461 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.185998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.19.65.188:40259)
0619 10:21:56.186033 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.186035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.186043 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:56.186125 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.186125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.186134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.186188 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.186456 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.186915 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.187676 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.187879 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.187972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.189578 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.189579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.189580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.189586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.190866 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.191260 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.191261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.191261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.191341 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.192010 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.192011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.192011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.192099 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.192175 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.192324 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.192386 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:21:56.192464 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.185948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.0.0.1:49502)
0619 10:21:56.185995 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.186003 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.186039 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.186192 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.186193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.186338 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.186374 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.186375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.186994 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.186995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.187578 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.187580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.187958 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.187959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.188023 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.188424 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.188427 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.189498 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.190949 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.190949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.190950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.191179 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.191406 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.191407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.191408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.191464 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.192168 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.192169 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.192208 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.192279 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.192377 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:21:56.192855 25798 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.65.188:40259: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
member_type: NON_VOTER
last_known_addr {
host: "127.19.65.188"
port: 40259
}
attrs {
promote: true
}
}
dest_uuid: "f7acac245ba8476787c80a6bf7924059"
cas_config_opid_index: 5
I20260619 10:21:56.193028 25798 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } }
I20260619 10:21:56.193352 26359 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.194402 26359 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.65.188:40259
I20260619 10:21:56.194481 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.194536 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.194725 26493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:56.196074 26493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 10:21:56.196448 26357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:56.197157 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.193449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60694)
0619 10:21:56.193713 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.193722 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.193753 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.193955 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.193956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.193957 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.193984 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.193985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.194849 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.194850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.196148 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.196151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.196929 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.196930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.196985 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.197005 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.197064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.197096 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260619 10:21:56.197158 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.193529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.19.65.189:39195)
0619 10:21:56.193689 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.193691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.193814 (+ 123us) server_negotiation.cc:407] Connection header received
0619 10:21:56.193835 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.193836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.193858 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.193907 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.194056 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.194706 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.196242 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.196833 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.196928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.197006 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.197070 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.197100 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260619 10:21:56.197484 26404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 LEARNER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 10:21:56.197662 26357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 10:21:56.198184 26404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.199103 26451 master.cc:537] Master@127.19.65.188:40259 shutting down...
I20260619 10:21:56.199271 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7acac245ba8476787c80a6bf7924059. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.199402 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: LEARNER
I20260619 10:21:56.199445 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: NON_VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: true } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.199494 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: LEARNER
I20260619 10:21:56.199525 26357 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: attempting to promote NON_VOTER ce218024e9854da0ad769ecd889cb7ea to VOTER
I20260619 10:21:56.199703 26357 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } }
I20260619 10:21:56.200069 26404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 LEARNER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 10:21:56.200336 26359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 10:21:56.200611 25783 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 -> Peer 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188:40259): Couldn't send request to peer 6efcfa1b1e4f4e83bc2aa6d0051e0c19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 10:21:56.201046 26357 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ce218024e9854da0ad769ecd889cb7ea (127.19.65.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.201198 26404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ce218024e9854da0ad769ecd889cb7ea (127.19.65.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.201942 26357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.202028 26357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.202106 26357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.202160 26357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.202287 26546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.202384 26546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:21:56.202945 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:44023
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:46391
--webserver_port=33283
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:21:56.202966 26451 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:56.203048 26451 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:56.203097 26451 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: stopping tablet replica
I20260619 10:21:56.206269 26451 master.cc:552] Master@127.19.65.188:40259 shutdown complete.
I20260619 10:21:56.206328 26451 master_runner.cc:305] Clearing existing system tablet
I20260619 10:21:56.206465 26451 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.206521 26451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.207140 26451 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.207618 26550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.207741 26451 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.207783 26451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.207813 26451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
format_stamp: "Formatted at 2026-06-19 10:21:56 on dist-test-slave-xd2q"
server_key: "3c543bda46c7238ba6f0f2b4d0dc9aac"
server_key_iv: "6356b900b37b68eafc8ec48dd3848158"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.208371 26451 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:56.209537 26451 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 10:21:56.209612 26451 log.cc:1196] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 10:21:56.209815 26451 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Deleting consensus metadata
I20260619 10:21:56.209946 26451 master_runner.cc:315] Copying system tablet from 127.19.65.190:46391
I20260619 10:21:56.210541 26451 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: tablet copy: Beginning tablet copy session from remote peer at address 127.19.65.190:46391
I20260619 10:21:56.216648 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.210811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42404)
0619 10:21:56.210917 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.210929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.210983 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.211105 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.211106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.211271 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.211422 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.211424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.212078 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.212079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.212619 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.212621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.213000 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.213000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.213074 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.213511 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.213516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.213804 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.215590 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.215591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.215592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.215773 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.215972 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.215973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.215974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.216049 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.216299 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.216300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.216340 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.216419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.216541 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":69,"threads_started":1}
I20260619 10:21:56.216630 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.210836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.19.65.190:46391)
0619 10:21:56.210924 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.210926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.210969 (+ 43us) server_negotiation.cc:407] Connection header received
0619 10:21:56.211038 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.211039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.211048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.211100 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.211501 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.211998 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.212719 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.212919 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.212998 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.213905 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.213907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.213907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.213914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.215507 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.215854 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.215855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.215856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.215907 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.216097 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.216098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.216098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.216209 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.216285 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.216462 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.216537 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 10:21:56.217077 25818 tablet_copy_service.cc:140] P f7acac245ba8476787c80a6bf7924059: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6efcfa1b1e4f4e83bc2aa6d0051e0c19 ({username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:42404)
I20260619 10:21:56.217155 25818 tablet_copy_service.cc:161] P f7acac245ba8476787c80a6bf7924059: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6efcfa1b1e4f4e83bc2aa6d0051e0c19 at {username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:42404: session id = 6efcfa1b1e4f4e83bc2aa6d0051e0c19-00000000000000000000000000000000
I20260619 10:21:56.217569 25818 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 10:21:56.218024 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 10:21:56.218899 26451 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: tablet copy: Starting download of 0 data blocks...
I20260619 10:21:56.219048 26451 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: tablet copy: Starting download of 1 WAL segments...
I20260619 10:21:56.219718 26451 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 10:21:56.220404 25818 tablet_copy_service.cc:342] P f7acac245ba8476787c80a6bf7924059: Request end of tablet copy session 6efcfa1b1e4f4e83bc2aa6d0051e0c19-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.65.188@KRBTEST.COM'} at 127.0.0.1:42404
I20260619 10:21:56.220454 25818 tablet_copy_service.cc:433] P f7acac245ba8476787c80a6bf7924059: ending tablet copy session 6efcfa1b1e4f4e83bc2aa6d0051e0c19-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6efcfa1b1e4f4e83bc2aa6d0051e0c19
I20260619 10:21:56.222645 26451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:56.223778 26557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.228060 26558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.228852 26560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.229048 26451 server_base.cc:1034] running on GCE node
I20260619 10:21:56.229138 26451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:56.229264 26451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:56.230392 26451 hybrid_clock.cc:648] HybridClock initialized: now 1781864516230388 us; error 24 us; skew 500 ppm
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:56.231725 26451 init.cc:377] Logged in from keytab as oryx/127.19.65.188@KRBTEST.COM (short username oryx)
I20260619 10:21:56.231997 26451 webserver.cc:460] Webserver started at http://127.19.65.188:45889/ using document root <none> and password file <none>
I20260619 10:21:56.232110 26451 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.232144 26451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.232745 26451 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.233220 26567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.233357 26451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.233402 26451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
format_stamp: "Formatted at 2026-06-19 10:21:56 on dist-test-slave-xd2q"
server_key: "3c543bda46c7238ba6f0f2b4d0dc9aac"
server_key_iv: "6356b900b37b68eafc8ec48dd3848158"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.233464 26451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.241317 26451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:56.242924 26451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:56.245965 26451 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:40259
I20260619 10:21:56.246251 26619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.188:40259 every 8 connection(s)
I20260619 10:21:56.246351 26451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
I20260619 10:21:56.246999 26620 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:21:56.247512 26620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Bootstrap starting.
I20260619 10:21:56.249593 26620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:21:56.249882 26620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: Bootstrap complete.
I20260619 10:21:56.250087 26620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } }
I20260619 10:21:56.250168 26620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6efcfa1b1e4f4e83bc2aa6d0051e0c19, State: Initialized, Role: LEARNER
I20260619 10:21:56.250206 26620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } }
I20260619 10:21:56.250375 26620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:21:56.250356 26623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.250560 26623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: LEARNER
I20260619 10:21:56.251801 26634 catalog_manager.cc:1259] Loaded cluster ID: a2a65fd65a594d9c973ad0cf77c8103e
I20260619 10:21:56.252148 26634 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: loading cluster ID for follower catalog manager: success
I20260619 10:21:56.252923 26634 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: acquiring CA information for follower catalog manager: success
I20260619 10:21:56.253176 26634 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 10:21:56.311704 26548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:56.311893 26548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:56.311916 26548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:56.313040 26548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:56.313078 26548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:56.313104 26548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:21:56.314535 26548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=33283
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:56.314754 26548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:56.315016 26548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:56.321135 26639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.321197 26640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.321198 26642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.321429 26548 server_base.cc:1034] running on GCE node
I20260619 10:21:56.321624 26548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:56.321866 26548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:56.323006 26548 hybrid_clock.cc:648] HybridClock initialized: now 1781864516322989 us; error 36 us; skew 500 ppm
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:56.325369 26548 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20260619 10:21:56.325803 26548 webserver.cc:460] Webserver started at http://127.19.65.129:33283/ using document root <none> and password file <none>
I20260619 10:21:56.326026 26548 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.326102 26548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.327419 26548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:56.327971 26649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.328168 26548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.328246 26548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "089ee58cdbcd41c3a15dd017674b5482"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "8ee13e3c3d10bfb774365b959622c892"
server_key_iv: "6495da2b7cbdf7ce9bdac2ceadbf3ef2"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.328517 26548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.340323 26548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:56.341888 26548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:56.341981 26548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:56.342137 26548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:56.342370 26548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:56.342396 26548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.342415 26548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:56.342427 26548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.347891 26548 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:44023
I20260619 10:21:56.347926 26762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:44023 every 8 connection(s)
I20260619 10:21:56.348269 26548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:21:56.348529 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26548
I20260619 10:21:56.348671 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25994
I20260619 10:21:56.356137 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:33029
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:46391
--webserver_port=42995
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20260619 10:21:56.361513 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.349333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45045 (local address 127.19.65.190:46391)
0619 10:21:56.349389 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.349391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.349760 (+ 369us) server_negotiation.cc:407] Connection header received
0619 10:21:56.350058 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.350060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.350072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.350102 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.351149 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.351941 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353496 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353783 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353833 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.358152 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.358154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.358154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.358181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.360293 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360716 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.360783 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360979 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.361113 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.361151 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.361354 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.361429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:21:56.361575 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.349233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:60325 (local address 127.19.65.189:39195)
0619 10:21:56.349440 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.349442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.349688 (+ 246us) server_negotiation.cc:407] Connection header received
0619 10:21:56.350054 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.350055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.350068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.350099 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.351149 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.351965 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353493 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353838 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353881 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.358127 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.358128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.358128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.358136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.359905 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360409 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.360471 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360762 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.360934 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.360983 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.361364 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.361447 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260619 10:21:56.362010 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.349330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53455 (local address 127.19.65.188:40259)
0619 10:21:56.349449 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.349451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.349463 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:21:56.350092 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.350094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.350105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.350186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.351147 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.352164 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353491 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353784 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353837 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.358124 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.358126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.358126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.358133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.360153 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360587 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.360647 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360863 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.360864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.360864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.360994 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.361063 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.361888 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.361950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260619 10:21:56.362253 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.348928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.19.65.129:53455)
0619 10:21:56.349347 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.349403 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.349947 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.350249 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.350253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.350755 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.351044 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.351050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.352241 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.352241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.353399 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.353401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353892 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.354035 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.354718 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.354725 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.358044 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.360251 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360508 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.360724 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360793 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.361086 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.361087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.361209 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.361310 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.361455 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":82,"thread_start_us":84,"threads_started":1}
I20260619 10:21:56.362253 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.349179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:45045)
0619 10:21:56.349714 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.349721 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.349957 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.350249 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.350253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.350755 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.351044 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.351050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.352054 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.352055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.353399 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.353401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353904 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.354035 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.354790 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.354804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.358050 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.360369 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360634 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.360859 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360910 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.361205 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.361205 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.361235 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.361598 (+ 363us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.361697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":100,"thread_start_us":58,"threads_started":1}
I20260619 10:21:56.363018 26763 heartbeater.cc:347] Connected to a master server at 127.19.65.188:40259
I20260619 10:21:56.363107 26763 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.363260 26763 heartbeater.cc:511] Master 127.19.65.188:40259 requested a full tablet report, sending...
I20260619 10:21:56.363519 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.348929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.19.65.129:60325)
0619 10:21:56.349603 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.349613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.349959 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.350249 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.350253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.350788 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.351044 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.351050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.352062 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.352063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.353398 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.353401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.353961 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.353962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.354063 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.354935 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.354940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.358015 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.359994 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360312 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.360547 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.360548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.360549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.360683 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.361036 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.361055 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.361211 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.361732 (+ 521us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.361858 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":83,"thread_start_us":78,"threads_started":1}
I20260619 10:21:56.363590 26764 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:56.363763 26584 ts_manager.cc:194] Registered new tserver with Master: 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
I20260619 10:21:56.363832 26764 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.363914 26764 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:56.364162 26765 heartbeater.cc:347] Connected to a master server at 127.19.65.189:39195
I20260619 10:21:56.364207 26765 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.364245 26765 heartbeater.cc:511] Master 127.19.65.189:39195 requested a full tablet report, sending...
I20260619 10:21:56.364251 25798 ts_manager.cc:194] Re-registered known tserver with Master: 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
I20260619 10:21:56.364779 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:45045
I20260619 10:21:56.365257 26397 ts_manager.cc:194] Registered new tserver with Master: 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
W20260619 10:21:56.437148 26771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:56.437361 26771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:56.437451 26771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:56.438643 26771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:56.438696 26771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:56.438737 26771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:21:56.440178 26771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:33029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=42995
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:56.440421 26771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:56.440623 26771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:56.445343 26776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.445415 26771 server_base.cc:1034] running on GCE node
W20260619 10:21:56.445477 26777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.445477 26779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.445760 26771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:56.445986 26771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:56.447146 26771 hybrid_clock.cc:648] HybridClock initialized: now 1781864516447140 us; error 41 us; skew 500 ppm
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:56.449502 26771 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20260619 10:21:56.449980 26771 webserver.cc:460] Webserver started at http://127.19.65.130:42995/ using document root <none> and password file <none>
I20260619 10:21:56.450196 26771 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.450270 26771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.451768 26771 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:21:56.452337 26786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.452498 26771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.452584 26771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "d9389632ff074715883a8444f9286359"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "bf7d54bdf6c3719d457325313ad063ad"
server_key_iv: "0acae512d2f0ae9cc7b0446fc93f0301"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.452872 26771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.468420 26771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:56.470129 26771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:56.470274 26771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:56.470484 26771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:56.470767 26771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:56.470820 26771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.470856 26771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:56.470883 26771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.476446 26771 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:33029
I20260619 10:21:56.476497 26899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:33029 every 8 connection(s)
I20260619 10:21:56.476797 26771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:21:56.481952 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26771
I20260619 10:21:56.482096 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26128
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
I20260619 10:21:56.489120 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.478124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:53469 (local address 127.19.65.188:40259)
0619 10:21:56.478205 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.478207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.478216 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:56.479212 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.479213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.479230 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.479306 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.479916 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.480636 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.482642 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.482981 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.483029 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.486132 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.486133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.486133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.486140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.487756 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.488148 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.488149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.488150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.488223 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.488474 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.488475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.488475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.488589 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.488659 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.488941 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.489033 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 10:21:56.489327 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.477741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.19.65.130:53469)
0619 10:21:56.477981 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.477991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.478801 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.479321 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.479322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.479594 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.479797 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.479799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.481793 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.481793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.482544 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.482547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.483105 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.483106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.483190 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.483691 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.483696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.486031 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.487837 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.487843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.487845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.488040 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.488283 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.488284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.488284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.488407 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.488651 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.488657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.488791 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.488911 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.489059 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
I20260619 10:21:56.489365 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:37843
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:46391
--webserver_port=46441
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:21:56.490422 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.477878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60909 (local address 127.19.65.189:39195)
0619 10:21:56.477921 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.477923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.477933 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:56.478923 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.478925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.478938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.478992 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.479884 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.480390 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.481798 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.482094 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.482183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.486131 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.486132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.486133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.486140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.488105 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489506 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.489507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.489508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.489579 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489767 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.489768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.489768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.489896 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.489939 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.490144 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.490241 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:21:56.490758 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.477513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.19.65.130:60909)
0619 10:21:56.477878 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.477888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.478807 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.479062 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.479066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.479588 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.479789 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.479794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.480492 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.480493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.481696 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.481698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.482223 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.482224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.482355 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.482895 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.482904 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.486031 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.489156 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.489158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.489402 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.489645 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.489646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.489703 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.489982 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.489995 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.490044 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.490502 (+ 458us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.490620 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260619 10:21:56.491153 26902 heartbeater.cc:347] Connected to a master server at 127.19.65.189:39195
I20260619 10:21:56.491281 26902 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.491264 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.478044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51441 (local address 127.19.65.190:46391)
0619 10:21:56.478104 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.478106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.478117 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:56.479051 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.479053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.479064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.479119 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.479916 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.480648 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.482571 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.482903 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.482962 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.485620 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.485622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.485622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.485630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.489068 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489952 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.489954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.489954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.490027 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.490239 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.490240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.490241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.490364 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.490457 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.491039 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.491129 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:21:56.491493 26902 heartbeater.cc:511] Master 127.19.65.189:39195 requested a full tablet report, sending...
I20260619 10:21:56.491153 26900 heartbeater.cc:347] Connected to a master server at 127.19.65.188:40259
I20260619 10:21:56.491634 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.477435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:51441)
0619 10:21:56.477649 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.477687 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.478791 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.479133 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.479133 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.479603 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.479790 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.479795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.480713 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.480713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.481695 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.481698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.483012 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.483013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.483085 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.483669 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.483674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.485527 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.489688 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.489689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.489689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.489853 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.490091 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.490092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.490093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.490146 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.490448 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.490448 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.490704 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.491397 (+ 693us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.491510 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":15,"thread_start_us":101,"threads_started":1}
I20260619 10:21:56.491676 26900 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.491742 26900 heartbeater.cc:511] Master 127.19.65.188:40259 requested a full tablet report, sending...
I20260619 10:21:56.491920 26397 ts_manager.cc:194] Registered new tserver with Master: d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:56.492081 26584 ts_manager.cc:194] Registered new tserver with Master: d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:56.492254 26901 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:56.492301 26901 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.492357 26901 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:56.492569 25798 ts_manager.cc:194] Re-registered known tserver with Master: d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:56.493083 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:51441
W20260619 10:21:56.569303 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:56.569478 26907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:56.569499 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:56.570554 26907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:56.570585 26907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:56.570609 26907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:21:56.572038 26907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:37843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=46441
--tserver_master_addrs=127.19.65.190:46391,127.19.65.189:39195,127.19.65.188:40259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:56.572199 26907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:56.572360 26907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:56.577028 26915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.577106 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:56.577042 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:56.577353 26907 server_base.cc:1034] running on GCE node
I20260619 10:21:56.577548 26907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:56.577725 26907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:56.578855 26907 hybrid_clock.cc:648] HybridClock initialized: now 1781864516578836 us; error 33 us; skew 500 ppm
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:56.581295 26907 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20260619 10:21:56.581708 26907 webserver.cc:460] Webserver started at http://127.19.65.131:46441/ using document root <none> and password file <none>
I20260619 10:21:56.581915 26907 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:56.581991 26907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:56.583365 26907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:56.583969 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.584139 26907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:56.584244 26907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "dfe9df95a5ef4c83a0384fbe432ec39d"
format_stamp: "Formatted at 2026-06-19 10:21:55 on dist-test-slave-xd2q"
server_key: "4b22ac8e5d0e74c9e0543548693f45d7"
server_key_iv: "e963decf23c9d60047f2444c7b05abfc"
server_key_version: "encryptionkey@0"
I20260619 10:21:56.584654 26907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:21:56.600474 26907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:21:56.602092 26907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:21:56.602239 26907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:21:56.602458 26907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:21:56.602727 26907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:21:56.602779 26907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.602819 26907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:21:56.602845 26907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:56.608449 26907 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:37843
I20260619 10:21:56.608492 27035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:37843 every 8 connection(s)
I20260619 10:21:56.608786 26907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:21:56.614970 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 26907
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.188@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
Jun 19 10:21:56 dist-test-slave-xd2q krb5kdc[25753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864516, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.189@KRBTEST.COM
I20260619 10:21:56.620685 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:39131 (local address 127.19.65.188:40259)
0619 10:21:56.609478 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.609480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.609841 (+ 361us) server_negotiation.cc:407] Connection header received
0619 10:21:56.610745 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.610746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.610760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.610846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.611524 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612008 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613003 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.613198 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613284 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.617385 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.617386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.617386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.617394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.619142 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.619622 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.619624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.619624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.619686 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.619994 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.619995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.619995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.620130 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.620181 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.620508 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.620596 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:56.621179 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.19.65.131:39131)
0619 10:21:56.609695 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.609731 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.610638 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.610913 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.610917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.611191 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.611437 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.611442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612112 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.612113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.612911 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.612913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.613291 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.613425 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.613919 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.613927 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.617299 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.619229 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.619236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.619238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.619534 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.619761 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.619761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.619762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.619910 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.620215 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.620220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.620360 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.620804 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.620965 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":210,"threads_started":1}
I20260619 10:21:56.621527 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:46279)
0619 10:21:56.609756 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.609764 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.610656 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.610913 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.610918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.611257 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.611438 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.611444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612398 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.612398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.613325 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.613326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.613751 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.613814 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.614228 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.614233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.616743 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.620261 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.620262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.620660 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.620876 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.620878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.620942 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.621207 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.621208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.621248 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.621302 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.621403 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":140,"thread_start_us":183,"threads_started":1}
I20260619 10:21:56.621596 27036 heartbeater.cc:347] Connected to a master server at 127.19.65.188:40259
I20260619 10:21:56.621687 27036 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.621906 27036 heartbeater.cc:511] Master 127.19.65.188:40259 requested a full tablet report, sending...
I20260619 10:21:56.621924 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46243 (local address 127.19.65.189:39195)
0619 10:21:56.609621 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.609622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.609826 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:21:56.610756 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.610758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.610771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.610814 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.611548 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612301 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613810 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.614075 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.614158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.616787 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.616799 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.616800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.616807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.620168 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620741 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.620742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.620742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.620795 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.621084 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.621084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.621085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.621201 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.621255 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.621760 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.621844 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:21:56.621920 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:46279 (local address 127.19.65.190:46391)
0619 10:21:56.609529 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.609531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.609821 (+ 290us) server_negotiation.cc:407] Connection header received
0619 10:21:56.610757 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.610758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.610771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.610804 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.611548 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612303 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613388 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.613672 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.613753 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.616832 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.616833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.616834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.616841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.619213 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620741 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.620742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.620742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.620794 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.621007 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.621008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.621008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.621129 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.621183 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.621735 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.621810 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:21:56.622375 26584 ts_manager.cc:194] Registered new tserver with Master: dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843)
I20260619 10:21:56.622532 27037 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46391
I20260619 10:21:56.622583 27037 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.622644 27037 heartbeater.cc:511] Master 127.19.65.190:46391 requested a full tablet report, sending...
I20260619 10:21:56.622810 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.609398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.19.65.131:46243)
0619 10:21:56.609748 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.609759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.610658 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.610913 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.610918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.611251 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:21:56.611438 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.611444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.612398 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.612399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.613318 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.613320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.614216 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.614217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.614276 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.614644 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:21:56.614650 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:21:56.616690 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:21:56.620256 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.620258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.620661 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.620960 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:21:56.620961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:21:56.620961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:21:56.621016 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:21:56.621392 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:21:56.621393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:21:56.621425 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.621485 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.621563 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":59,"thread_start_us":186,"threads_started":1}
I20260619 10:21:56.622846 25798 ts_manager.cc:194] Re-registered known tserver with Master: dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843)
I20260619 10:21:56.623360 25798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:46279
I20260619 10:21:56.623641 27038 heartbeater.cc:347] Connected to a master server at 127.19.65.189:39195
I20260619 10:21:56.623683 27038 heartbeater.cc:464] Registering TS with master...
I20260619 10:21:56.623734 27038 heartbeater.cc:511] Master 127.19.65.189:39195 requested a full tablet report, sending...
I20260619 10:21:56.624305 26397 ts_manager.cc:194] Registered new tserver with Master: dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843)
I20260619 10:21:56.625931 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.616065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.19.65.190:46391)
0619 10:21:56.616506 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.616507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.616516 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:56.616535 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.616536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.616547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.616981 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.617852 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.618567 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.619319 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.619639 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.619950 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.622929 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.622930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.622931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.622937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.624691 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.625120 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.625121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.625122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.625177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.625411 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.625411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.625412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.625518 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.625616 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.625729 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.625820 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":80,"threads_started":1}
I20260619 10:21:56.627266 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.616201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.19.65.189:39195)
0619 10:21:56.616432 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.616434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.616525 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:21:56.616614 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.616616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.616628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.616667 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.618294 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.619958 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.622229 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.622581 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.622869 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.623985 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.623986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.623987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.623993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.625738 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.626222 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.626223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.626224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.626278 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.626551 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.626552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.626552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.626663 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.627056 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.627095 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.627172 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":54,"threads_started":1}
I20260619 10:21:56.627993 25798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:56.629993 25798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:56.631376 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.617894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.19.65.188:40259)
0619 10:21:56.620335 (+ 2441us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.620337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.620464 (+ 127us) server_negotiation.cc:407] Connection header received
0619 10:21:56.622753 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.622755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.622769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.624154 (+ 1385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.625090 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.625852 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.626823 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.627117 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.627174 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.628743 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.628744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.628744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.628751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.630139 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.630477 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.630478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.630478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.630526 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.630786 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.630787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.630787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.630909 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.630999 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.631133 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.631211 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2420,"thread_start_us":266,"threads_started":1}
I20260619 10:21:56.637475 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.633772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33029 (local address 127.0.0.1:38244)
0619 10:21:56.633907 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.633917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.634115 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.634381 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.634382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.634384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.634424 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.634425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.635308 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.635309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.636431 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.636433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.637236 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.637298 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.637303 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.637370 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.637407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260619 10:21:56.637719 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.633699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:37843 (local address 127.0.0.1:35610)
0619 10:21:56.633793 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.633804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.633843 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.634331 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.634331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.634333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.634369 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.634371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.635588 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.635589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.636669 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.636670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.637540 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.637596 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.637599 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.637626 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.637659 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 10:21:56.637732 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.633872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.19.65.130:33029)
0619 10:21:56.634070 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.634074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.634085 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:21:56.634213 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.634215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.634231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.634285 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.634506 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.635205 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.636512 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.637137 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637233 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.637306 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.637383 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.637420 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260619 10:21:56.638079 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.633916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.19.65.131:37843)
0619 10:21:56.634155 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.634159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.634167 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:56.634198 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.634201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.634215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.634319 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.634421 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.635496 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.636759 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.637452 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637833 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.637900 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.637963 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.638003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
I20260619 10:21:56.638324 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.633799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44023 (local address 127.0.0.1:55416)
0619 10:21:56.634518 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.634526 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.634567 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.634807 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.634807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.634809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.634841 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.634842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.636067 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.636068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.637121 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.637123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.638090 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.638091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.638153 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.638157 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.638211 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.638253 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":59,"threads_started":1}
I20260619 10:21:56.638396 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.634091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.19.65.129:44023)
0619 10:21:56.634269 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.634273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.634620 (+ 347us) server_negotiation.cc:407] Connection header received
0619 10:21:56.634664 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.634666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.634681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.634745 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.634918 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.635984 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637205 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.637933 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.637994 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.638056 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.638267 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.638305 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260619 10:21:56.638506 26834 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00bf25a45447308ba48874e34493c9 (DEFAULT_TABLE table=test-table [id=4f7bf022ff4346bf9feb91d0f6a047c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:56.638850 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00bf25a45447308ba48874e34493c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:56.639003 26697 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00bf25a45447308ba48874e34493c9 (DEFAULT_TABLE table=test-table [id=4f7bf022ff4346bf9feb91d0f6a047c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:56.639387 26697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00bf25a45447308ba48874e34493c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:56.640022 26970 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00bf25a45447308ba48874e34493c9 (DEFAULT_TABLE table=test-table [id=4f7bf022ff4346bf9feb91d0f6a047c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:56.640395 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00bf25a45447308ba48874e34493c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:56.642455 27062 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Bootstrap starting.
I20260619 10:21:56.642951 27064 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Bootstrap starting.
I20260619 10:21:56.642940 27063 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Bootstrap starting.
I20260619 10:21:56.643254 27062 tablet_bootstrap.cc:654] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:56.643601 27062 log.cc:824] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:56.643633 27063 tablet_bootstrap.cc:654] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:56.643683 27064 tablet_bootstrap.cc:654] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:56.643960 27063 log.cc:824] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:56.643988 27064 log.cc:824] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:21:56.644696 27062 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: No bootstrap required, opened a new log
I20260619 10:21:56.644696 27064 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: No bootstrap required, opened a new log
I20260619 10:21:56.644766 27062 ts_tablet_manager.cc:1397] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:21:56.644766 27064 ts_tablet_manager.cc:1397] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:56.645056 27063 tablet_bootstrap.cc:492] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: No bootstrap required, opened a new log
I20260619 10:21:56.645138 27063 ts_tablet_manager.cc:1397] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:21:56.647100 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.639880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.19.65.188:40259)
0619 10:21:56.639911 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.639912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.643183 (+ 3271us) server_negotiation.cc:407] Connection header received
0619 10:21:56.643284 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.643285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.643336 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.643377 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.643601 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.644135 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.645932 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.646559 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.646622 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.646691 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.646957 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.647014 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:21:56.647110 27062 raft_consensus.cc:348] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.647274 27062 raft_consensus.cc:374] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:56.647222 27064 raft_consensus.cc:348] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.647352 27062 raft_consensus.cc:729] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9389632ff074715883a8444f9286359, State: Initialized, Role: FOLLOWER
I20260619 10:21:56.647382 27064 raft_consensus.cc:374] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:56.647420 27064 raft_consensus.cc:729] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe9df95a5ef4c83a0384fbe432ec39d, State: Initialized, Role: FOLLOWER
I20260619 10:21:56.647483 27062 consensus_queue.cc:260] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.647522 27064 consensus_queue.cc:260] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.647750 27064 ts_tablet_manager.cc:1428] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:21:56.647750 27062 ts_tablet_manager.cc:1428] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 10:21:56.647843 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.639760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.0.0.1:49512)
0619 10:21:56.643130 (+ 3370us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.643141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.643207 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.643492 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.643493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.643495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.643524 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.643525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.644366 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.644367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.645810 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.645813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.646770 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.646771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.646852 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.646855 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.647506 (+ 651us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.647558 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3353}
I20260619 10:21:56.647804 27063 raft_consensus.cc:348] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.647970 27063 raft_consensus.cc:374] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:56.648005 27063 raft_consensus.cc:729] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ee58cdbcd41c3a15dd017674b5482, State: Initialized, Role: FOLLOWER
I20260619 10:21:56.648165 27037 heartbeater.cc:503] Master 127.19.65.190:46391 was elected leader, sending a full tablet report...
I20260619 10:21:56.648145 27063 consensus_queue.cc:260] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.648402 27063 ts_tablet_manager.cc:1428] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 10:21:56.648487 26764 heartbeater.cc:503] Master 127.19.65.190:46391 was elected leader, sending a full tablet report...
I20260619 10:21:56.648571 26594 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 10:21:56.649588 26623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7acac245ba8476787c80a6bf7924059. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.649688 26623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: LEARNER
I20260619 10:21:56.651121 26901 heartbeater.cc:503] Master 127.19.65.190:46391 was elected leader, sending a full tablet report...
W20260619 10:21:56.727532 26903 tablet.cc:2369] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:56.849887 26766 tablet.cc:2369] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:21:56.859817 27039 tablet.cc:2369] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:21:56.904675 27070 raft_consensus.cc:482] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:56.904784 27070 raft_consensus.cc:504] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.905228 27070 leader_election.cc:288] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843), d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:56.908106 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.905470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:56717 (local address 127.19.65.130:33029)
0619 10:21:56.905513 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.905515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.905730 (+ 215us) server_negotiation.cc:407] Connection header received
0619 10:21:56.905757 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.905758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.905769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.905826 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.905983 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.906423 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907393 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.907777 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907831 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.907861 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.907998 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.908036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:21:56.908141 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.905407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:60191 (local address 127.19.65.131:37843)
0619 10:21:56.905452 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.905455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.905610 (+ 155us) server_negotiation.cc:407] Connection header received
0619 10:21:56.905645 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.905646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.905658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.905715 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:56.905891 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.906407 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907446 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.907775 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907872 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.907880 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.908036 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.908072 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:21:56.908169 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.905343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:37843 (local address 127.19.65.129:60191)
0619 10:21:56.905514 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.905525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.905590 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.905750 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.905751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.905753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.905815 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.905817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.906491 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.906492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.907360 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.907362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.907891 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.907944 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.907947 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.908019 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.908058 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":25,"thread_start_us":60,"threads_started":1}
I20260619 10:21:56.908352 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.905394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33029 (local address 127.19.65.129:56717)
0619 10:21:56.905630 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:56.905641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:56.905704 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:56.905839 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:56.905840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:56.905841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:56.905917 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.905919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.906534 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.906535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.907269 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:56.907271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.907853 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.907854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:56.907921 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.907927 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:21:56.907975 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 10:21:56.908031 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2,"thread_start_us":44,"threads_started":1}
I20260619 10:21:56.908469 26990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00bf25a45447308ba48874e34493c9" candidate_uuid: "089ee58cdbcd41c3a15dd017674b5482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" is_pre_election: true
I20260619 10:21:56.908617 26990 raft_consensus.cc:2454] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089ee58cdbcd41c3a15dd017674b5482 in term 0.
I20260619 10:21:56.908888 26653 leader_election.cc:302] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089ee58cdbcd41c3a15dd017674b5482, dfe9df95a5ef4c83a0384fbe432ec39d; no voters:
I20260619 10:21:56.908900 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00bf25a45447308ba48874e34493c9" candidate_uuid: "089ee58cdbcd41c3a15dd017674b5482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9389632ff074715883a8444f9286359" is_pre_election: true
I20260619 10:21:56.909013 27070 raft_consensus.cc:2781] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:56.909025 26854 raft_consensus.cc:2454] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089ee58cdbcd41c3a15dd017674b5482 in term 0.
I20260619 10:21:56.909073 27070 raft_consensus.cc:482] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:56.909090 27070 raft_consensus.cc:3037] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:56.909807 27070 raft_consensus.cc:504] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.909940 27070 leader_election.cc:288] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [CANDIDATE]: Term 1 election: Requested vote from peers dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843), d9389632ff074715883a8444f9286359 (127.19.65.130:33029)
I20260619 10:21:56.910133 26990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00bf25a45447308ba48874e34493c9" candidate_uuid: "089ee58cdbcd41c3a15dd017674b5482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d"
I20260619 10:21:56.910140 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00bf25a45447308ba48874e34493c9" candidate_uuid: "089ee58cdbcd41c3a15dd017674b5482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9389632ff074715883a8444f9286359"
I20260619 10:21:56.910213 26990 raft_consensus.cc:3037] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:56.910250 26854 raft_consensus.cc:3037] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:56.910929 26854 raft_consensus.cc:2454] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089ee58cdbcd41c3a15dd017674b5482 in term 1.
I20260619 10:21:56.910939 26990 raft_consensus.cc:2454] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089ee58cdbcd41c3a15dd017674b5482 in term 1.
I20260619 10:21:56.911135 26651 leader_election.cc:302] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089ee58cdbcd41c3a15dd017674b5482, d9389632ff074715883a8444f9286359; no voters:
I20260619 10:21:56.911227 27070 raft_consensus.cc:2781] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:56.911342 27070 raft_consensus.cc:686] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 1 LEADER]: Becoming Leader. State: Replica: 089ee58cdbcd41c3a15dd017674b5482, State: Running, Role: LEADER
I20260619 10:21:56.911448 27070 consensus_queue.cc:237] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:56.912161 25798 catalog_manager.cc:5495] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 reported cstate change: term changed from 0 to 1, leader changed from <none> to 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "089ee58cdbcd41c3a15dd017674b5482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } health_report { overall_health: HEALTHY } } }
I20260619 10:21:56.912789 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: NON_VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: true } } }
I20260619 10:21:56.912887 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.913396 26359 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: attempting to promote NON_VOTER 6efcfa1b1e4f4e83bc2aa6d0051e0c19 to VOTER
I20260619 10:21:56.913522 26359 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } }
I20260619 10:21:56.913928 26404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 10:21:56.913928 26594 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 LEARNER]: Refusing update from remote peer f7acac245ba8476787c80a6bf7924059: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 10:21:56.914116 26359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260619 10:21:56.914250 26358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260619 10:21:56.914971 26357 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.915186 26594 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.915889 26357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.915915 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.916003 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.916055 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.916118 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.916045 26404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.916211 26623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.916281 26623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:21:56.915967 26357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059 [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:56.917443 27080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7acac245ba8476787c80a6bf7924059" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:56.917508 27080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:21:56.920487 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.918411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.19.65.129:44023)
0619 10:21:56.918444 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.918446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.918454 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:56.918520 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.918521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.918533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.918588 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:56.918709 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.919130 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.919907 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.920075 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.920125 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.920263 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:56.920344 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:56.920398 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.920430 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.920445 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:21:56.921922 26854 raft_consensus.cc:1261] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Refusing update from remote peer 089ee58cdbcd41c3a15dd017674b5482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:56.921959 26990 raft_consensus.cc:1261] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Refusing update from remote peer 089ee58cdbcd41c3a15dd017674b5482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:56.922276 27077 consensus_queue.cc:1035] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:56.922410 27070 consensus_queue.cc:1035] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:56.925858 27082 mvcc.cc:205] Tried to move back new op lower bound from 7298517061310185472 to 7298517061269823488. Current Snapshot: MvccSnapshot[applied={T|T < 7298517061310185472}]
I20260619 10:21:56.926764 27083 mvcc.cc:205] Tried to move back new op lower bound from 7298517061310185472 to 7298517061269823488. Current Snapshot: MvccSnapshot[applied={T|T < 7298517061310185472}]
I20260619 10:21:56.928423 25798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:56.928504 25798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:56.929977 25798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7acac245ba8476787c80a6bf7924059: Sending DeleteTablet for 3 replicas of tablet 8d00bf25a45447308ba48874e34493c9
I20260619 10:21:56.930429 26970 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d00bf25a45447308ba48874e34493c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:56 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35610
I20260619 10:21:56.930440 26697 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d00bf25a45447308ba48874e34493c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:56 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55416
I20260619 10:21:56.930634 27092 tablet_replica.cc:330] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: stopping tablet replica
I20260619 10:21:56.930732 27092 raft_consensus.cc:2229] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:56.930823 26834 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d00bf25a45447308ba48874e34493c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:56 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:38244
I20260619 10:21:56.931046 27092 raft_consensus.cc:2258] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:56.931211 27094 tablet_replica.cc:330] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: stopping tablet replica
I20260619 10:21:56.931022 27093 tablet_replica.cc:330] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: stopping tablet replica
I20260619 10:21:56.931320 27094 raft_consensus.cc:2229] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:56.931414 27094 raft_consensus.cc:2258] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:56.931605 27092 ts_tablet_manager.cc:1905] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:56.931806 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 25766
I20260619 10:21:56.932421 27094 ts_tablet_manager.cc:1905] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:56.933962 27094 ts_tablet_manager.cc:1918] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:56.934041 27094 log.cc:1196] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/8d00bf25a45447308ba48874e34493c9
I20260619 10:21:56.934109 27092 ts_tablet_manager.cc:1918] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:56.934183 27092 log.cc:1196] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/8d00bf25a45447308ba48874e34493c9
I20260619 10:21:56.934315 27094 ts_tablet_manager.cc:1939] T 8d00bf25a45447308ba48874e34493c9 P d9389632ff074715883a8444f9286359: Deleting consensus metadata
I20260619 10:21:56.934437 27092 ts_tablet_manager.cc:1939] T 8d00bf25a45447308ba48874e34493c9 P 089ee58cdbcd41c3a15dd017674b5482: Deleting consensus metadata
W20260619 10:21:56.938349 26653 connection.cc:535] server connection from 127.0.0.1:55416 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 10:21:56.938455 26787 connection.cc:535] server connection from 127.0.0.1:38244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:21:56.939388 27093 raft_consensus.cc:2229] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:56.939505 27093 raft_consensus.cc:2258] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:56.939898 27093 ts_tablet_manager.cc:1905] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:56.941443 27093 ts_tablet_manager.cc:1918] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:56.941515 27093 log.cc:1196] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/8d00bf25a45447308ba48874e34493c9
I20260619 10:21:56.941780 27093 ts_tablet_manager.cc:1939] T 8d00bf25a45447308ba48874e34493c9 P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting consensus metadata
I20260619 10:21:56.941948 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.931260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.19.65.188:40259)
0619 10:21:56.931341 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.931343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.935463 (+ 4120us) server_negotiation.cc:407] Connection header received
0619 10:21:56.935533 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.935534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.935544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.935600 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.936176 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.936704 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.937370 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.937578 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.937653 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.939147 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.939148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.939148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.939155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.940866 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.941206 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.941207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.941208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.941261 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.941450 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.941451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.941451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.941578 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.941671 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.941766 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.941844 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:21:56.942096 26926 connection.cc:441] server connection from 127.0.0.1:35610 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35610 (request call id 1) could send its response
I20260619 10:21:56.950649 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.943772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.19.65.188:40259)
0619 10:21:56.943806 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.943808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.943926 (+ 118us) server_negotiation.cc:407] Connection header received
0619 10:21:56.944005 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.944018 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.944025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.944051 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.944719 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.945386 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.946063 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.946327 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.946381 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.947793 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.947794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.947794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.947800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.949376 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.949767 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.949769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.949769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.949821 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.950063 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.950064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.950064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.950196 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.950280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.950458 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.950539 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:21:56.951009 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:56.943563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.19.65.189:39195)
0619 10:21:56.943609 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:56.943612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:56.943721 (+ 109us) server_negotiation.cc:407] Connection header received
0619 10:21:56.943821 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:56.943822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:56.943836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:56.943866 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:21:56.944526 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.945229 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.946211 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:56.946503 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:56.946590 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:56.947987 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:21:56.947988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:21:56.947989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:21:56.947996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:21:56.949534 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.949957 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.949957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.949958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.950008 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:21:56.950233 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:21:56.950235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:21:56.950235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:21:56.950390 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:21:56.950495 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:56.950824 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 10:21:56.950905 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:21:57.925966 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:57.925715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:40413)
0619 10:21:57.925846 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:57.925919 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260619 10:21:57.926151 26764 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:46391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:57.926721 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:57.926393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:52131)
0619 10:21:57.926567 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:57.926675 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":44,"threads_started":1}
W20260619 10:21:57.927002 27037 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:46391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:57.926997 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:57.926359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:40049)
0619 10:21:57.926891 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:57.926964 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":55,"threads_started":1}
W20260619 10:21:57.927182 26901 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:46391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:58.430572 27106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7acac245ba8476787c80a6bf7924059)
I20260619 10:21:58.430727 27106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } }
I20260619 10:21:58.430763 27107 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7acac245ba8476787c80a6bf7924059)
I20260619 10:21:58.430896 27107 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } }
I20260619 10:21:58.431198 27107 leader_election.cc:288] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391), 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188:40259)
I20260619 10:21:58.431211 27106 leader_election.cc:288] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391), ce218024e9854da0ad769ecd889cb7ea (127.19.65.189:39195)
I20260619 10:21:58.431463 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42882)
0619 10:21:58.431378 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.431424 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":44,"threads_started":1}
I20260619 10:21:58.431666 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42898)
0619 10:21:58.431577 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.431626 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":48,"threads_started":1}
W20260619 10:21:58.431768 26384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
W20260619 10:21:58.431769 26571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:58.432231 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.432155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42912)
0619 10:21:58.432190 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.432211 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:21:58.432252 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.432178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42918)
0619 10:21:58.432209 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.432229 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 10:21:58.432327 26384 leader_election.cc:334] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391): Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
W20260619 10:21:58.432440 26571 leader_election.cc:334] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391): Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:58.434340 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.19.65.188:40259)
0619 10:21:58.431783 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.431785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.431793 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:21:58.431814 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.431814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.431823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.432039 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.432084 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.432781 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.433650 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.433995 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.434080 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.434091 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.434221 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.434258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":41,"threads_started":1}
I20260619 10:21:58.434369 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:40259 (local address 127.0.0.1:49532)
0619 10:21:58.431635 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.431643 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.431695 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.431908 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.431908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.431910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.431964 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.431966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.432850 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.432851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.433584 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.433587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.434083 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.434084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.434144 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.434147 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.434214 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.434254 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":45,"threads_started":1}
I20260619 10:21:58.434475 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.19.65.189:39195)
0619 10:21:58.431416 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.431418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.431520 (+ 102us) server_negotiation.cc:407] Connection header received
0619 10:21:58.431625 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.431626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.431635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.431707 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.431945 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.432708 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.433703 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.434045 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.434119 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.434129 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.434255 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.434294 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":179,"thread_start_us":61,"threads_started":1}
I20260619 10:21:58.434576 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.431140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:39195 (local address 127.0.0.1:60714)
0619 10:21:58.431461 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.431472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.431545 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.431796 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.431797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.431800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.431860 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.431863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.432777 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.432778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.433623 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.433625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.434120 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.434121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.434181 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.434184 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.434241 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.434280 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":47,"threads_started":1}
I20260619 10:21:58.434608 26594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce218024e9854da0ad769ecd889cb7ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" is_pre_election: true
I20260619 10:21:58.434715 26594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce218024e9854da0ad769ecd889cb7ea in term 1.
I20260619 10:21:58.434898 26382 leader_election.cc:302] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6efcfa1b1e4f4e83bc2aa6d0051e0c19, ce218024e9854da0ad769ecd889cb7ea; no voters: f7acac245ba8476787c80a6bf7924059
I20260619 10:21:58.435046 27107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 10:21:58.435091 27107 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7acac245ba8476787c80a6bf7924059)
I20260619 10:21:58.435122 27107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:21:58.435279 26404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce218024e9854da0ad769ecd889cb7ea" is_pre_election: true
I20260619 10:21:58.435897 27107 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } }
I20260619 10:21:58.436017 26404 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6efcfa1b1e4f4e83bc2aa6d0051e0c19 in current term 2: Already voted for candidate ce218024e9854da0ad769ecd889cb7ea in this term.
I20260619 10:21:58.436033 27107 leader_election.cc:288] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 election: Requested vote from peers f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391), 6efcfa1b1e4f4e83bc2aa6d0051e0c19 (127.19.65.188:40259)
I20260619 10:21:58.436159 26594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce218024e9854da0ad769ecd889cb7ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19"
I20260619 10:21:58.436232 26594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:21:58.436235 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.436102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42924)
0619 10:21:58.436184 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.436213 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 10:21:58.436254 26568 leader_election.cc:302] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6efcfa1b1e4f4e83bc2aa6d0051e0c19; no voters: ce218024e9854da0ad769ecd889cb7ea, f7acac245ba8476787c80a6bf7924059
I20260619 10:21:58.436620 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.436506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42938)
0619 10:21:58.436563 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.436593 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 10:21:58.436702 26384 leader_election.cc:334] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391): Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
I20260619 10:21:58.436986 26594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce218024e9854da0ad769ecd889cb7ea in term 2.
I20260619 10:21:58.437062 27106 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:21:58.437161 26382 leader_election.cc:302] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6efcfa1b1e4f4e83bc2aa6d0051e0c19, ce218024e9854da0ad769ecd889cb7ea; no voters: f7acac245ba8476787c80a6bf7924059
I20260619 10:21:58.437260 27107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:21:58.437392 27107 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [term 2 LEADER]: Becoming Leader. State: Replica: ce218024e9854da0ad769ecd889cb7ea, State: Running, Role: LEADER
I20260619 10:21:58.437484 27107 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } }
I20260619 10:21:58.437852 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce218024e9854da0ad769ecd889cb7ea. Latest consensus state: current_term: 2 leader_uuid: "ce218024e9854da0ad769ecd889cb7ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:58.437950 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:58.438092 27118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:21:58.438439 27118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:21:58.438491 27118 catalog_manager.cc:1259] Loaded cluster ID: a2a65fd65a594d9c973ad0cf77c8103e
I20260619 10:21:58.438539 27118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:21:58.438709 27118 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:21:58.438794 27118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Loaded TSK: 0
I20260619 10:21:58.438930 27118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:21:58.479709 26396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:21:58.481267 26396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:21:58.482272 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.482142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42970)
0619 10:21:58.482220 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.482246 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 10:21:58.482268 26594 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [term 2 FOLLOWER]: Refusing update from remote peer ce218024e9854da0ad769ecd889cb7ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 10:21:58.482509 27116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 10:21:58.482659 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.482553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.0.0.1:42978)
0619 10:21:58.482603 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.482631 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 10:21:58.482808 26384 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea -> Peer f7acac245ba8476787c80a6bf7924059 (127.19.65.190:46391): Couldn't send request to peer f7acac245ba8476787c80a6bf7924059. Status: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 10:21:58.483173 27106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce218024e9854da0ad769ecd889cb7ea. Latest consensus state: current_term: 2 leader_uuid: "ce218024e9854da0ad769ecd889cb7ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:58.483264 27106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:21:58.483624 27119 mvcc.cc:205] Tried to move back new op lower bound from 7298517067701555200 to 7298517067520737280. Current Snapshot: MvccSnapshot[applied={T|T < 7298517067701555200}]
I20260619 10:21:58.483743 27107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ce218024e9854da0ad769ecd889cb7ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:58.483829 27107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea [sys.catalog]: This master's current role is: LEADER
I20260619 10:21:58.483946 27106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ce218024e9854da0ad769ecd889cb7ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7acac245ba8476787c80a6bf7924059" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46391 } } peers { permanent_uuid: "ce218024e9854da0ad769ecd889cb7ea" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 39195 } attrs { promote: false } } peers { permanent_uuid: "6efcfa1b1e4f4e83bc2aa6d0051e0c19" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 40259 } attrs { promote: false } } }
I20260619 10:21:58.484040 27106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfa1b1e4f4e83bc2aa6d0051e0c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:21:58.484236 27120 mvcc.cc:205] Tried to move back new op lower bound from 7298517067701555200 to 7298517067520737280. Current Snapshot: MvccSnapshot[applied={T|T < 7298517067701555200 or (T in {7298517067701555200})}]
I20260619 10:21:58.488771 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33029 (local address 127.0.0.1:38250)
0619 10:21:58.485371 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.485380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.485414 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.485765 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.485766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.485768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.485819 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.485821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.486654 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.486655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.487809 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.487823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488568 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.488628 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488633 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.488663 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.488705 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":21}
I20260619 10:21:58.489003 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.19.65.130:33029)
0619 10:21:58.485611 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.485613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.485622 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:21:58.485649 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.485650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.485661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.485879 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.485911 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.486578 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.487915 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488484 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488878 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488892 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.488909 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.488943 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260619 10:21:58.488986 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:37843 (local address 127.0.0.1:35626)
0619 10:21:58.485622 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.485631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.485716 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.485840 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.485841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.485843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.485917 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.485919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.486857 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.486858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.487998 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.487999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488744 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.488820 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488822 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.488873 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.488913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260619 10:21:58.489171 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.19.65.129:44023)
0619 10:21:58.486195 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.486197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.486204 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:21:58.486228 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.486229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.486239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.486300 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.486427 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.487042 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488234 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488671 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488757 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488768 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.489034 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.489085 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":51,"threads_started":1}
I20260619 10:21:58.489387 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.19.65.131:37843)
0619 10:21:58.485639 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.485641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.485703 (+ 62us) server_negotiation.cc:407] Connection header received
0619 10:21:58.485767 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.485768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.485777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.485838 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.486003 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.486781 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488074 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488656 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488728 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488740 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.489282 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.489329 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260619 10:21:58.489392 26834 tablet_service.cc:1467] Processing CreateTablet for tablet 29812d44e12a44a5871d2d7a06b0157d (DEFAULT_TABLE table=test-table [id=0f7c336473264000ab6132c094a77584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:58.489518 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29812d44e12a44a5871d2d7a06b0157d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:58.489524 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.485418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44023 (local address 127.0.0.1:55420)
0619 10:21:58.485995 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.486004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.486050 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.486313 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.486314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.486316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.486357 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.486358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.487155 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.487156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.488132 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.488134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.488793 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.488794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.488913 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.488915 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.488954 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.488983 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":451,"thread_start_us":66,"threads_started":1}
I20260619 10:21:58.489562 26970 tablet_service.cc:1467] Processing CreateTablet for tablet 29812d44e12a44a5871d2d7a06b0157d (DEFAULT_TABLE table=test-table [id=0f7c336473264000ab6132c094a77584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:58.489670 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29812d44e12a44a5871d2d7a06b0157d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:58.489722 26697 tablet_service.cc:1467] Processing CreateTablet for tablet 29812d44e12a44a5871d2d7a06b0157d (DEFAULT_TABLE table=test-table [id=0f7c336473264000ab6132c094a77584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:21:58.489822 26697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29812d44e12a44a5871d2d7a06b0157d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:21:58.490633 27129 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Bootstrap starting.
I20260619 10:21:58.490995 27129 tablet_bootstrap.cc:654] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:58.491268 27131 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482: Bootstrap starting.
I20260619 10:21:58.491314 27130 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Bootstrap starting.
I20260619 10:21:58.491648 27131 tablet_bootstrap.cc:654] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:58.491662 27130 tablet_bootstrap.cc:654] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Neither blocks nor log segments found. Creating new log.
I20260619 10:21:58.492187 27130 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: No bootstrap required, opened a new log
I20260619 10:21:58.492254 27130 ts_tablet_manager.cc:1397] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:21:58.492252 27129 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: No bootstrap required, opened a new log
I20260619 10:21:58.492304 27129 ts_tablet_manager.cc:1397] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:21:58.492353 27131 tablet_bootstrap.cc:492] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482: No bootstrap required, opened a new log
I20260619 10:21:58.492401 27131 ts_tablet_manager.cc:1397] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:21:58.492373 27130 raft_consensus.cc:348] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492446 27130 raft_consensus.cc:374] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:58.492425 27129 raft_consensus.cc:348] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492475 27130 raft_consensus.cc:729] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe9df95a5ef4c83a0384fbe432ec39d, State: Initialized, Role: FOLLOWER
I20260619 10:21:58.492491 27129 raft_consensus.cc:374] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:58.492514 27129 raft_consensus.cc:729] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9389632ff074715883a8444f9286359, State: Initialized, Role: FOLLOWER
I20260619 10:21:58.492511 27130 consensus_queue.cc:260] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492511 27131 raft_consensus.cc:348] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492547 27129 consensus_queue.cc:260] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492566 27131 raft_consensus.cc:374] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:21:58.492592 27131 raft_consensus.cc:729] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ee58cdbcd41c3a15dd017674b5482, State: Initialized, Role: FOLLOWER
I20260619 10:21:58.492635 27130 ts_tablet_manager.cc:1428] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:58.492627 27131 consensus_queue.cc:260] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.492669 27129 ts_tablet_manager.cc:1428] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:58.492761 27131 ts_tablet_manager.cc:1428] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:21:58.493304 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.492971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:53983)
0619 10:21:58.493079 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.493112 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":72,"threads_started":1}
I20260619 10:21:58.493671 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.493495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:59659)
0619 10:21:58.493604 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.493637 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":73,"threads_started":1}
I20260619 10:21:58.493985 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.493200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:38935)
0619 10:21:58.493914 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.493948 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":68,"threads_started":1}
I20260619 10:21:58.494275 27038 heartbeater.cc:503] Master 127.19.65.189:39195 was elected leader, sending a full tablet report...
I20260619 10:21:58.494472 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.494224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:55089)
0619 10:21:58.494258 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.494287 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 10:21:58.494824 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.494571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:44587)
0619 10:21:58.494773 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.494799 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 10:21:58.495050 26902 heartbeater.cc:503] Master 127.19.65.189:39195 was elected leader, sending a full tablet report...
I20260619 10:21:58.495160 26765 heartbeater.cc:503] Master 127.19.65.189:39195 was elected leader, sending a full tablet report...
I20260619 10:21:58.495275 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.495088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:43671)
0619 10:21:58.495171 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.495242 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 10:21:58.659360 27135 raft_consensus.cc:482] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:21:58.659452 27135 raft_consensus.cc:504] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.659942 27135 leader_election.cc:288] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9389632ff074715883a8444f9286359 (127.19.65.130:33029), 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
I20260619 10:21:58.662739 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.660185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:54547 (local address 127.19.65.130:33029)
0619 10:21:58.660229 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.660231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.660241 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:21:58.660347 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.660348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.660362 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.660422 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.660570 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.661051 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662013 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.662377 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662470 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.662480 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.662640 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.662683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:21:58.662731 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.660112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33029 (local address 127.19.65.131:54547)
0619 10:21:58.660181 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.660193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.660256 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.660429 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.660430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.660432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.660489 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.660495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.661146 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.661147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.661910 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.661912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.662468 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.662544 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.662546 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.662627 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.662664 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 10:21:58.662941 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29812d44e12a44a5871d2d7a06b0157d" candidate_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9389632ff074715883a8444f9286359" is_pre_election: true
I20260619 10:21:58.663030 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.660283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40457 (local address 127.19.65.129:44023)
0619 10:21:58.660333 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.660336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.660445 (+ 109us) server_negotiation.cc:407] Connection header received
0619 10:21:58.660542 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.660543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.660552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.660628 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:21:58.660788 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.661261 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662204 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.662583 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662686 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.662697 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.662911 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.662963 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:21:58.663048 26854 raft_consensus.cc:2454] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe9df95a5ef4c83a0384fbe432ec39d in term 0.
I20260619 10:21:58.663271 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.660155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44023 (local address 127.19.65.131:40457)
0619 10:21:58.660348 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.660358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.660455 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.660639 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:21:58.660640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:21:58.660641 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:21:58.660710 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.660711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.661364 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.661365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.662098 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:21:58.662100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.662708 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.662709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:21:58.662802 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.662807 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:21:58.663128 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 10:21:58.663188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":125,"threads_started":1}
I20260619 10:21:58.663256 26924 leader_election.cc:302] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9389632ff074715883a8444f9286359, dfe9df95a5ef4c83a0384fbe432ec39d; no voters:
I20260619 10:21:58.663470 27135 raft_consensus.cc:2781] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:21:58.663511 27135 raft_consensus.cc:482] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:21:58.663537 27135 raft_consensus.cc:3037] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:58.663637 26717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29812d44e12a44a5871d2d7a06b0157d" candidate_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089ee58cdbcd41c3a15dd017674b5482" is_pre_election: true
I20260619 10:21:58.663764 26717 raft_consensus.cc:2454] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe9df95a5ef4c83a0384fbe432ec39d in term 0.
I20260619 10:21:58.664407 27135 raft_consensus.cc:504] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.664566 27135 leader_election.cc:288] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [CANDIDATE]: Term 1 election: Requested vote from peers d9389632ff074715883a8444f9286359 (127.19.65.130:33029), 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023)
I20260619 10:21:58.664733 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29812d44e12a44a5871d2d7a06b0157d" candidate_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9389632ff074715883a8444f9286359"
I20260619 10:21:58.664752 26717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29812d44e12a44a5871d2d7a06b0157d" candidate_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089ee58cdbcd41c3a15dd017674b5482"
I20260619 10:21:58.664810 26854 raft_consensus.cc:3037] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:58.664829 26717 raft_consensus.cc:3037] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:21:58.665632 26854 raft_consensus.cc:2454] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe9df95a5ef4c83a0384fbe432ec39d in term 1.
I20260619 10:21:58.665717 26717 raft_consensus.cc:2454] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe9df95a5ef4c83a0384fbe432ec39d in term 1.
I20260619 10:21:58.665774 26924 leader_election.cc:302] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9389632ff074715883a8444f9286359, dfe9df95a5ef4c83a0384fbe432ec39d; no voters:
I20260619 10:21:58.665858 27135 raft_consensus.cc:2781] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:21:58.665984 27135 raft_consensus.cc:686] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 LEADER]: Becoming Leader. State: Replica: dfe9df95a5ef4c83a0384fbe432ec39d, State: Running, Role: LEADER
I20260619 10:21:58.666039 27135 consensus_queue.cc:237] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } }
I20260619 10:21:58.666347 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.666246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:54159)
0619 10:21:58.666286 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.666316 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:21:58.666870 26397 catalog_manager.cc:5495] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131). New cstate: current_term: 1 leader_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe9df95a5ef4c83a0384fbe432ec39d" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 } health_report { overall_health: UNKNOWN } } }
I20260619 10:21:58.714305 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.712268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.19.65.131:37843)
0619 10:21:58.712306 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:21:58.712308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:21:58.712314 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:21:58.712380 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:21:58.712380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:21:58.712395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:21:58.712449 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:21:58.712541 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.712981 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.713642 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:21:58.713814 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:21:58.713865 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:21:58.713982 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:21:58.714073 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:21:58.714138 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:21:58.714188 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 10:21:58.714203 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:21:58.715386 26854 raft_consensus.cc:1261] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Refusing update from remote peer dfe9df95a5ef4c83a0384fbe432ec39d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:58.715557 26717 raft_consensus.cc:1261] T 29812d44e12a44a5871d2d7a06b0157d P 089ee58cdbcd41c3a15dd017674b5482 [term 1 FOLLOWER]: Refusing update from remote peer dfe9df95a5ef4c83a0384fbe432ec39d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:21:58.715683 27135 consensus_queue.cc:1035] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9389632ff074715883a8444f9286359" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:58.715816 27142 consensus_queue.cc:1035] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [LEADER]: Connected to new peer: Peer: permanent_uuid: "089ee58cdbcd41c3a15dd017674b5482" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:21:58.716691 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.716582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:45739)
0619 10:21:58.716626 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.716656 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:21:58.716867 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.716769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:34223)
0619 10:21:58.716808 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.716833 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:21:58.717841 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.717735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:45939)
0619 10:21:58.717775 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.717800 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:21:58.718160 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.718021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.130:58631)
0619 10:21:58.718111 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.718135 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 10:21:58.719267 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.719188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.129:44839)
0619 10:21:58.719216 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.719242 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260619 10:21:58.719949 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.719810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46391 (local address 127.19.65.131:52617)
0619 10:21:58.719900 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.719924 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 10:21:58.719985 26396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:58.720117 26396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:21:58.720332 27146 mvcc.cc:205] Tried to move back new op lower bound from 7298517068656013312 to 7298517068457234432. Current Snapshot: MvccSnapshot[applied={T|T < 7298517068656013312}]
I20260619 10:21:58.721283 26396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce218024e9854da0ad769ecd889cb7ea: Sending DeleteTablet for 3 replicas of tablet 29812d44e12a44a5871d2d7a06b0157d
I20260619 10:21:58.721566 26970 tablet_service.cc:1514] Processing DeleteTablet for tablet 29812d44e12a44a5871d2d7a06b0157d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:58 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:35626
I20260619 10:21:58.721621 26697 tablet_service.cc:1514] Processing DeleteTablet for tablet 29812d44e12a44a5871d2d7a06b0157d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:58 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:55420
I20260619 10:21:58.721737 27155 tablet_replica.cc:330] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: stopping tablet replica
I20260619 10:21:58.721782 27155 raft_consensus.cc:2229] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:21:58.721887 27155 raft_consensus.cc:2258] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:58.721913 26834 tablet_service.cc:1514] Processing DeleteTablet for tablet 29812d44e12a44a5871d2d7a06b0157d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:21:58 UTC) from {username='oryx', principal='oryx/127.19.65.189@KRBTEST.COM'} at 127.0.0.1:38250
I20260619 10:21:58.722086 27157 tablet_replica.cc:330] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: stopping tablet replica
I20260619 10:21:58.722177 27157 raft_consensus.cc:2229] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:21:58.722191 27155 ts_tablet_manager.cc:1905] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:58.722277 27157 raft_consensus.cc:2258] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:21:58.722342 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26548
I20260619 10:21:58.724431 27155 ts_tablet_manager.cc:1918] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:58.724480 27155 log.cc:1196] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/29812d44e12a44a5871d2d7a06b0157d
I20260619 10:21:58.724706 27155 ts_tablet_manager.cc:1939] T 29812d44e12a44a5871d2d7a06b0157d P dfe9df95a5ef4c83a0384fbe432ec39d: Deleting consensus metadata
I20260619 10:21:58.725010 26384 catalog_manager.cc:4841] TS dfe9df95a5ef4c83a0384fbe432ec39d (127.19.65.131:37843): tablet 29812d44e12a44a5871d2d7a06b0157d (table test-table [id=0f7c336473264000ab6132c094a77584]) successfully deleted
I20260619 10:21:58.727605 27157 ts_tablet_manager.cc:1905] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:21:58.729009 27157 ts_tablet_manager.cc:1918] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:21:58.729068 27157 log.cc:1196] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/29812d44e12a44a5871d2d7a06b0157d
W20260619 10:21:58.729266 27128 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:21:58.729056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44023 (local address 127.0.0.1:55428)
0619 10:21:58.729094 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.729104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:21:58.729138 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:21:58.729219 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:44023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:21:58.729285 27157 ts_tablet_manager.cc:1939] T 29812d44e12a44a5871d2d7a06b0157d P d9389632ff074715883a8444f9286359: Deleting consensus metadata
I20260619 10:21:58.729517 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26771
W20260619 10:21:58.731663 26383 catalog_manager.cc:4584] TS 089ee58cdbcd41c3a15dd017674b5482 (127.19.65.129:44023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29812d44e12a44a5871d2d7a06b0157d: Network error: Client connection negotiation failed: client connection to 127.19.65.129:44023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:21:58.735143 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:21:58.735048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:33029 (local address 127.0.0.1:38252)
0619 10:21:58.735085 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 10:21:58.735111 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.130:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 10:21:58.735260 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26907
I20260619 10:21:58.740752 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26262
I20260619 10:21:58.745285 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 26451
2026-06-19T10:21:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4866 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:21:58 dist-test-slave-xd2q krb5kdc[27160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:21:58 dist-test-slave-xd2q krb5kdc[27160](info): set up 2 sockets
Jun 19 10:21:58 dist-test-slave-xd2q krb5kdc[27160](info): commencing operation
krb5kdc: starting...
W20260619 10:21:59.787647 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:21:59 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:21:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab.
Entry for principal oryx/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab.
I20260619 10:21:59.897384 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35213
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab
--principal=oryx/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:35213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:21:59.974413 27173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:21:59.974644 27173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:21:59.974679 27173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:21:59.975871 27173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:21:59.975950 27173 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:21:59.975972 27173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:21:59.975996 27173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:21:59.976024 27173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:21:59.976048 27173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:21:59.977492 27173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:35213
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.190.keytab
--principal=oryx/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:21:59.977752 27173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:21:59.977969 27173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:21:59.982586 27181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:59.982652 27173 server_base.cc:1034] running on GCE node
W20260619 10:21:59.982625 27179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:21:59.982638 27178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:21:59.983058 27173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:21:59.983281 27173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:21:59.984478 27173 hybrid_clock.cc:648] HybridClock initialized: now 1781864519984470 us; error 29 us; skew 500 ppm
Jun 19 10:21:59 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864519, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:21:59.987226 27173 init.cc:377] Logged in from keytab as oryx/127.19.65.190@KRBTEST.COM (short username oryx)
I20260619 10:21:59.987643 27173 webserver.cc:460] Webserver started at http://127.19.65.190:44595/ using document root <none> and password file <none>
I20260619 10:21:59.987809 27173 fs_manager.cc:359] Metadata directory not provided
I20260619 10:21:59.987847 27173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:21:59.987929 27173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:21:59.988821 27173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "4be480d2c51b4bed9f66781666db6b77"
format_stamp: "Formatted at 2026-06-19 10:21:59 on dist-test-slave-xd2q"
server_key: "54d20357a18eeb68ca8da892e3f9030a"
server_key_iv: "4312c83a06062a3c0a60046df816bbb6"
server_key_version: "encryptionkey@0"
I20260619 10:21:59.989094 27173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "4be480d2c51b4bed9f66781666db6b77"
format_stamp: "Formatted at 2026-06-19 10:21:59 on dist-test-slave-xd2q"
server_key: "54d20357a18eeb68ca8da892e3f9030a"
server_key_iv: "4312c83a06062a3c0a60046df816bbb6"
server_key_version: "encryptionkey@0"
I20260619 10:21:59.990284 27173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:21:59.990974 27188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:21:59.991123 27173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:21:59.991210 27173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "4be480d2c51b4bed9f66781666db6b77"
format_stamp: "Formatted at 2026-06-19 10:21:59 on dist-test-slave-xd2q"
server_key: "54d20357a18eeb68ca8da892e3f9030a"
server_key_iv: "4312c83a06062a3c0a60046df816bbb6"
server_key_version: "encryptionkey@0"
I20260619 10:21:59.991264 27173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:00.028556 27173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:00.031190 27173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:00.031384 27173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:00.036125 27173 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:35213
I20260619 10:22:00.036211 27250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:35213 every 8 connection(s)
I20260619 10:22:00.036569 27173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:00.037331 27251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.039812 27251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Bootstrap starting.
I20260619 10:22:00.040405 27251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.040686 27251 log.cc:824] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:00.041373 27251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: No bootstrap required, opened a new log
I20260619 10:22:00.041662 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27173
I20260619 10:22:00.041751 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:22:00.041954 19718 external_mini_cluster.cc:1428] Setting key 7ef8297d8ba4c142e0a782b8c9d32920
I20260619 10:22:00.044102 27251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } }
I20260619 10:22:00.044237 27251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.044263 27251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be480d2c51b4bed9f66781666db6b77, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.044404 27251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } }
I20260619 10:22:00.044479 27251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:00.044502 27251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:00.044564 27251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:00.045070 27251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } }
I20260619 10:22:00.045153 27251 leader_election.cc:302] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be480d2c51b4bed9f66781666db6b77; no voters:
I20260619 10:22:00.045357 27251 leader_election.cc:288] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:22:00.045396 27256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:00.045511 27256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [term 1 LEADER]: Becoming Leader. State: Replica: 4be480d2c51b4bed9f66781666db6b77, State: Running, Role: LEADER
I20260619 10:22:00.045589 27256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } }
I20260619 10:22:00.045624 27251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:00.046126 27258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be480d2c51b4bed9f66781666db6b77. Latest consensus state: current_term: 1 leader_uuid: "4be480d2c51b4bed9f66781666db6b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } } }
I20260619 10:22:00.046200 27258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:00.046356 27257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be480d2c51b4bed9f66781666db6b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be480d2c51b4bed9f66781666db6b77" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35213 } } }
I20260619 10:22:00.046415 27257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:00.046739 27262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:00.047254 27262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:22:00.049387 27262 catalog_manager.cc:1347] Generated new cluster ID: e3b4fd7907424cd98934e367041dcda4
I20260619 10:22:00.049450 27262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:00.055371 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.042547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.19.65.190:35213)
0619 10:22:00.042720 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.042745 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.042776 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:22:00.044119 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.044130 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.044200 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.044411 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.045292 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.046695 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.047922 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.048604 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.048751 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.051685 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.051689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.051689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.051701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.053672 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.054035 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.054038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.054038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.054138 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.054324 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.054325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.054325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.054602 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.054732 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.054982 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.055117 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260619 10:22:00.074630 27262 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:22:00.075224 27262 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:00.088341 27262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Generated new TSK 0
I20260619 10:22:00.088579 27262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
I20260619 10:22:00.117846 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35213
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:00.217048 27277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:00.217243 27277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:00.217267 27277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:00.218349 27277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:00.218384 27277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:00.218403 27277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:22:00.218420 27277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:00.219871 27277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:00.220041 27277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:00.220216 27277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:00.225185 27283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:00.225358 27277 server_base.cc:1034] running on GCE node
W20260619 10:22:00.225287 27285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:00.225474 27282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:00.225669 27277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:00.225919 27277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:00.227094 27277 hybrid_clock.cc:648] HybridClock initialized: now 1781864520227080 us; error 45 us; skew 500 ppm
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:00.229581 27277 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20260619 10:22:00.230093 27277 webserver.cc:460] Webserver started at http://127.19.65.129:35249/ using document root <none> and password file <none>
I20260619 10:22:00.230329 27277 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:00.230402 27277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:00.230566 27277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:00.231721 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "e26d0c143aea498ba7ca09e22c2bd066"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "b688787492b6f3c0c1f7728a07a8ed74"
server_key_iv: "d726a083daec882200c8c4cb32074457"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.232090 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "e26d0c143aea498ba7ca09e22c2bd066"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "b688787492b6f3c0c1f7728a07a8ed74"
server_key_iv: "d726a083daec882200c8c4cb32074457"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.233443 27277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:00.234198 27292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.234377 27277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:00.234479 27277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "e26d0c143aea498ba7ca09e22c2bd066"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "b688787492b6f3c0c1f7728a07a8ed74"
server_key_iv: "d726a083daec882200c8c4cb32074457"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.234567 27277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:00.241989 27277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:00.244160 27277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:00.244330 27277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:00.244872 27277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:00.244938 27277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.244971 27277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:00.244995 27277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:00.253031 27277 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:46111
I20260619 10:22:00.253057 27407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:46111 every 8 connection(s)
I20260619 10:22:00.253429 27277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:22:00.254500 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.245239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37357 (local address 127.19.65.190:35213)
0619 10:22:00.245296 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.245298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.245498 (+ 200us) server_negotiation.cc:407] Connection header received
0619 10:22:00.245990 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.245991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.246006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.246069 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.246790 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.247350 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.248394 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.248629 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.248708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.251213 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.251215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.251215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.251224 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.253139 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.253635 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.253636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.253636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.253703 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.253979 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.253980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.253980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.254101 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.254169 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.254373 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.254438 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:22:00.254679 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.245227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.129:37357)
0619 10:22:00.245390 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.245421 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.245888 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.246132 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.246136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.246456 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.246692 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.246697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.247423 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.247424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.248299 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.248301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.248716 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.248717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.248848 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.249328 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.249337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.251097 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.253244 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.253249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.253251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.253549 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.253791 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.253793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.253793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.253917 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.254170 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.254174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.254281 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.254357 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.254466 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260619 10:22:00.255175 27408 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35213
I20260619 10:22:00.255272 27408 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:00.255489 27408 heartbeater.cc:511] Master 127.19.65.190:35213 requested a full tablet report, sending...
I20260619 10:22:00.255985 27204 ts_manager.cc:194] Registered new tserver with Master: e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111)
I20260619 10:22:00.256924 27204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:37357
I20260619 10:22:00.261255 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.256188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45587 (local address 127.19.65.190:35213)
0619 10:22:00.256226 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.256228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.256370 (+ 142us) server_negotiation.cc:407] Connection header received
0619 10:22:00.256431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.256432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.256440 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.256484 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.256819 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.257385 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.258058 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.258228 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.258285 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.259018 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.259019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.259019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.259024 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.260374 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.260688 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.260689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.260689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.260732 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.260897 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.260897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.260898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.260977 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.261020 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.261151 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.261203 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:22:00.261268 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.256139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.129:45587)
0619 10:22:00.256293 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.256304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.256369 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.256489 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.256490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.256695 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.256754 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.256756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.257482 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.257483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.257984 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.257985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.258299 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.258299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.258348 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.258675 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.258679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.258943 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.260446 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.260447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.260447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.260610 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.260792 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.260793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.260793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.260847 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.261033 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.261034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.261069 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.261137 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.261209 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260619 10:22:00.263019 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27277
I20260619 10:22:00.263154 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:22:00.263418 19718 external_mini_cluster.cc:1428] Setting key 9ca2525eb89cd9eaebdd58a02d82c75e
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
I20260619 10:22:00.287494 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35213
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:00.365438 27417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:00.365629 27417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:00.365651 27417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:00.366808 27417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:00.366849 27417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:00.366869 27417 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:22:00.366886 27417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:00.368330 27417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:00.368497 27417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:00.368651 27417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:22:00.373410 27417 server_base.cc:1034] running on GCE node
W20260619 10:22:00.373358 27425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:00.373350 27422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:00.373365 27423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:00.373723 27417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:00.373967 27417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:00.375131 27417 hybrid_clock.cc:648] HybridClock initialized: now 1781864520375109 us; error 40 us; skew 500 ppm
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:00.377621 27417 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20260619 10:22:00.378052 27417 webserver.cc:460] Webserver started at http://127.19.65.130:34745/ using document root <none> and password file <none>
I20260619 10:22:00.378279 27417 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:00.378346 27417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:00.378476 27417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:00.379536 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "2b3626e99ac64a1684327ecc1d7affc5"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "769bd233b7aafa4b0667a691b1fe0a14"
server_key_iv: "275f766441a6472d0f30ce97cf513da5"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.379877 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "2b3626e99ac64a1684327ecc1d7affc5"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "769bd233b7aafa4b0667a691b1fe0a14"
server_key_iv: "275f766441a6472d0f30ce97cf513da5"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.381132 27417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:00.381853 27432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.382023 27417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:00.382117 27417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "2b3626e99ac64a1684327ecc1d7affc5"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "769bd233b7aafa4b0667a691b1fe0a14"
server_key_iv: "275f766441a6472d0f30ce97cf513da5"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.382200 27417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:00.397513 27417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:00.399752 27417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:00.399909 27417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:00.400431 27417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:00.400485 27417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.400527 27417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:00.400549 27417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:00.407672 27417 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:46685
I20260619 10:22:00.407773 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:46685 every 8 connection(s)
I20260619 10:22:00.408042 27417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:22:00.410076 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.400792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:49359 (local address 127.19.65.190:35213)
0619 10:22:00.400860 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.400863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.401097 (+ 234us) server_negotiation.cc:407] Connection header received
0619 10:22:00.401654 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.401656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.401675 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.401760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.402364 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.403193 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.404322 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.404560 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.404641 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.407331 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.407332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.407333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.407342 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.408954 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.409348 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.409349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.409349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.409396 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.409603 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.409604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.409604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.409696 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.409743 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.409964 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.410017 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:22:00.410275 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.400796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.130:49359)
0619 10:22:00.400954 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.400990 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.401545 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.401824 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.401827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.402106 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.402284 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.402289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.403317 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.403318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.404191 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.404194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.404660 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.404661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.404801 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.405325 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.405342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.407213 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.409024 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.409028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.409029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.409279 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.409453 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.409454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.409454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.409549 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.409752 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.409756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.409859 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.409942 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.410054 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260619 10:22:00.410759 27548 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35213
I20260619 10:22:00.410841 27548 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:00.411015 27548 heartbeater.cc:511] Master 127.19.65.190:35213 requested a full tablet report, sending...
I20260619 10:22:00.411438 27204 ts_manager.cc:194] Registered new tserver with Master: 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685)
I20260619 10:22:00.411744 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27417
I20260619 10:22:00.411862 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:22:00.411943 27204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:49359
I20260619 10:22:00.412139 19718 external_mini_cluster.cc:1428] Setting key 5cb1f8199d80d0612c4d8cbb9bd4203e
I20260619 10:22:00.418082 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.411723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.130:59123)
0619 10:22:00.411955 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.411967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.412052 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.412247 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.412248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.412454 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.412518 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.412520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.413141 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.413142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.413896 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.413899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.414248 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.414249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.414323 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.414755 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.414761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.415072 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.417025 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.417026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.417026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.417186 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.417494 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.417494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.417494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.417542 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.417872 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.417873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.417905 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.417953 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.418019 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":146,"threads_started":1}
I20260619 10:22:00.418212 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.411784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59123 (local address 127.19.65.190:35213)
0619 10:22:00.411846 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.411848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.412026 (+ 178us) server_negotiation.cc:407] Connection header received
0619 10:22:00.412133 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.412134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.412145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.412179 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.412602 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.413043 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.413976 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.414174 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.414224 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.415187 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.415188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.415189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.415196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.416942 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.417343 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.417344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.417345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.417419 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.417605 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.417605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.417606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.417802 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.417884 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.417984 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.418068 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
I20260619 10:22:00.438692 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35213
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:00.518751 27557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:00.518931 27557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:00.518954 27557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:00.520069 27557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:00.520102 27557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:00.520121 27557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:22:00.520156 27557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:00.521538 27557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.65.190:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:00.521721 27557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:00.521872 27557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:22:00.526640 27557 server_base.cc:1034] running on GCE node
W20260619 10:22:00.526722 27562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:00.526612 27565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:00.526612 27563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:00.527022 27557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:00.527325 27557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:00.528471 27557 hybrid_clock.cc:648] HybridClock initialized: now 1781864520528469 us; error 32 us; skew 500 ppm
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:00.530871 27557 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20260619 10:22:00.531333 27557 webserver.cc:460] Webserver started at http://127.19.65.131:33487/ using document root <none> and password file <none>
I20260619 10:22:00.531550 27557 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:00.531615 27557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:00.531742 27557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:00.532794 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "bb45d156a268472b756dbfe6ae407456"
server_key_iv: "83e771c88b8d43175083f06cd412bc6a"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.533145 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "bb45d156a268472b756dbfe6ae407456"
server_key_iv: "83e771c88b8d43175083f06cd412bc6a"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.534408 27557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:22:00.535367 27572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.535545 27557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:00.535640 27557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
format_stamp: "Formatted at 2026-06-19 10:22:00 on dist-test-slave-xd2q"
server_key: "bb45d156a268472b756dbfe6ae407456"
server_key_iv: "83e771c88b8d43175083f06cd412bc6a"
server_key_version: "encryptionkey@0"
I20260619 10:22:00.535723 27557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:00.547410 27557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:00.549182 27557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:00.549335 27557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:00.549794 27557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:00.549856 27557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.549897 27557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:00.549925 27557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:00.556910 27557 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:46781
I20260619 10:22:00.556946 27687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:46781 every 8 connection(s)
I20260619 10:22:00.557267 27557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:22:00.560163 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.550254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:45953 (local address 127.19.65.190:35213)
0619 10:22:00.550306 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.550309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.550493 (+ 184us) server_negotiation.cc:407] Connection header received
0619 10:22:00.550955 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.550956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.550971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.551052 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.551767 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.552573 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.554235 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.554557 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.554635 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.557344 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.557346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.557346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.557354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.558819 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.559227 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.559228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.559228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.559331 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.559559 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.559560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.559560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.559673 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.559749 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.560000 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.560076 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:22:00.560339 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.550244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.131:45953)
0619 10:22:00.550374 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.550409 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.550835 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.551120 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.551124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.551487 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.551655 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.551659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.552694 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.552695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.554132 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.554136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.554665 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.554666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.554819 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.555328 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.555345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.557250 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.558918 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.558925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.558927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.559139 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.559382 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.559383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.559383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.559493 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.559740 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.559746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.559875 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.559978 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.560125 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260619 10:22:00.560827 27688 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35213
I20260619 10:22:00.560910 27688 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:00.561065 27688 heartbeater.cc:511] Master 127.19.65.190:35213 requested a full tablet report, sending...
I20260619 10:22:00.561488 27205 ts_manager.cc:194] Registered new tserver with Master: d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:00.561969 27205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:45953
I20260619 10:22:00.563453 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27557
I20260619 10:22:00.563561 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:22:00.563753 19718 external_mini_cluster.cc:1428] Setting key 916ffb7c88426d015f4795cc846a5e7c
I20260619 10:22:00.564767 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:22:00.568001 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.561832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:37201 (local address 127.19.65.190:35213)
0619 10:22:00.561868 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.561869 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.561929 (+ 60us) server_negotiation.cc:407] Connection header received
0619 10:22:00.561986 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.561987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.561994 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.562066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.562429 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.562881 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.563621 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.563890 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.563996 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.565022 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.565024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.565024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.565031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.566708 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.567159 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.567160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.567160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.567218 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.567473 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.567474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.567474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.567586 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.567666 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.567834 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.567909 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:22:00.568017 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.561752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.19.65.131:37201)
0619 10:22:00.561872 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.561882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.561936 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.562093 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.562093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.562293 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:00.562353 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.562354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.562971 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.562972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.563529 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.563531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.564033 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.564034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.564124 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.564585 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:00.564589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:00.564943 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:00.566818 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.566818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.566819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.567048 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.567321 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:00.567322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:00.567323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:00.567391 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:00.567656 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:00.567657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:00.567703 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.567808 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.567912 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:22:00.587241 27255 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:00.581759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.19.65.190:35213)
0619 10:22:00.581821 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.581822 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.581834 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:22:00.581896 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.581897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.581906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.581955 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.582523 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.582998 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.583791 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.583968 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.584023 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.587160 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34606: BlockingRecv error: recv got EOF from 127.0.0.1:34606 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 10:22:00 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:00.595697 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.588288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.19.65.190:35213)
0619 10:22:00.588348 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.588350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.588449 (+ 99us) server_negotiation.cc:407] Connection header received
0619 10:22:00.588528 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.588530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.588542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.588628 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:00.589162 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.589644 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.590342 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.590527 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.590619 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.592819 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:00.592821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:00.592821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:00.592828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:00.594397 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.594823 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.594824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.594825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.594903 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:00.595112 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:00.595113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:00.595113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:00.595236 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:00.595387 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.595480 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.595580 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:22:00.597461 27205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:22:00.598954 27205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:00.605305 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:46781 (local address 127.0.0.1:44038)
0619 10:22:00.601878 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.601888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.601936 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.602283 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.602284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.602286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:00.602322 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.602324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.603412 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.603414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.604303 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.604305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.605017 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.605093 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.605107 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.605179 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.605229 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":72,"threads_started":1}
I20260619 10:22:00.605324 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.19.65.131:46781)
0619 10:22:00.601712 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.601715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.602000 (+ 285us) server_negotiation.cc:407] Connection header received
0619 10:22:00.602120 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.602122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.602140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.602215 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:00.602403 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.603335 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.604393 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.604920 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605014 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.605091 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.605194 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.605229 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260619 10:22:00.605507 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.0.0.1:43602)
0619 10:22:00.601325 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.601368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.601438 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.601791 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.601794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.601796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:00.601878 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.601881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.602922 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.602923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.604310 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.604312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.605282 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.605353 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.605362 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.605395 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.605433 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":62,"thread_start_us":39,"threads_started":1}
I20260619 10:22:00.605834 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46111 (local address 127.0.0.1:42740)
0619 10:22:00.601443 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.601455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.601503 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.601811 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.601812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.601814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:00.601868 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.601870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.603089 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.603090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.604309 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.604312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.605282 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.605353 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.605360 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.605567 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.605764 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":6,"thread_start_us":103,"threads_started":1}
I20260619 10:22:00.605986 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.19.65.130:46685)
0619 10:22:00.601605 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.601609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.601640 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:22:00.601683 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.601686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.601702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.601789 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:00.601980 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.602838 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.604422 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.605165 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605642 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.605728 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.605758 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.605801 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":75,"threads_started":1}
I20260619 10:22:00.606138 27622 tablet_service.cc:1467] Processing CreateTablet for tablet 47d1bb9027d94763a506520ded135ae9 (DEFAULT_TABLE table=test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.606276 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.601371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.19.65.129:46111)
0619 10:22:00.601624 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.601628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.601638 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:00.601676 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.601678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.601692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.601810 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:00.601926 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.602994 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.604421 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.605181 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.605963 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.606048 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.606080 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.606136 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
I20260619 10:22:00.606510 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d1bb9027d94763a506520ded135ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.606570 27482 tablet_service.cc:1467] Processing CreateTablet for tablet 47d1bb9027d94763a506520ded135ae9 (DEFAULT_TABLE table=test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.606874 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d1bb9027d94763a506520ded135ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.606966 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 47d1bb9027d94763a506520ded135ae9 (DEFAULT_TABLE table=test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.607334 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d1bb9027d94763a506520ded135ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.610090 27716 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Bootstrap starting.
I20260619 10:22:00.610754 27718 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Bootstrap starting.
I20260619 10:22:00.610873 27716 tablet_bootstrap.cc:654] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.611186 27716 log.cc:824] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:00.611341 27718 tablet_bootstrap.cc:654] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.611446 27717 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Bootstrap starting.
I20260619 10:22:00.611651 27718 log.cc:824] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:00.612069 27716 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: No bootstrap required, opened a new log
I20260619 10:22:00.612079 27717 tablet_bootstrap.cc:654] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.612150 27716 ts_tablet_manager.cc:1397] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:22:00.612358 27717 log.cc:824] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:00.612380 27718 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: No bootstrap required, opened a new log
I20260619 10:22:00.612439 27718 ts_tablet_manager.cc:1397] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:22:00.613369 27717 tablet_bootstrap.cc:492] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: No bootstrap required, opened a new log
I20260619 10:22:00.613435 27717 ts_tablet_manager.cc:1397] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:22:00.613806 27716 raft_consensus.cc:348] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.613952 27716 raft_consensus.cc:374] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.613997 27716 raft_consensus.cc:729] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.614108 27716 consensus_queue.cc:260] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.614403 27716 ts_tablet_manager.cc:1428] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:22:00.614425 27408 heartbeater.cc:503] Master 127.19.65.190:35213 was elected leader, sending a full tablet report...
I20260619 10:22:00.614351 27718 raft_consensus.cc:348] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.614799 27718 raft_consensus.cc:374] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.614841 27718 raft_consensus.cc:729] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.614976 27718 consensus_queue.cc:260] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.615237 27718 ts_tablet_manager.cc:1428] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:22:00.615368 27548 heartbeater.cc:503] Master 127.19.65.190:35213 was elected leader, sending a full tablet report...
I20260619 10:22:00.615417 27717 raft_consensus.cc:348] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.615550 27717 raft_consensus.cc:374] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.615585 27717 raft_consensus.cc:729] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.615710 27717 consensus_queue.cc:260] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.615976 27717 ts_tablet_manager.cc:1428] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:22:00.616032 27688 heartbeater.cc:503] Master 127.19.65.190:35213 was elected leader, sending a full tablet report...
W20260619 10:22:00.658746 27549 tablet.cc:2369] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:22:00.754194 27409 tablet.cc:2369] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:22:00.768868 27722 raft_consensus.cc:482] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:00.768971 27722 raft_consensus.cc:504] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.769347 27722 leader_election.cc:288] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:00.772307 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.769636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:60825 (local address 127.19.65.130:46685)
0619 10:22:00.769691 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.769695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.769742 (+ 47us) server_negotiation.cc:407] Connection header received
0619 10:22:00.769825 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.769828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.769842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.769890 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:00.770099 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.770575 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.771619 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.771967 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.772061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.772069 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.772200 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.772233 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:22:00.772326 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.769835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44473 (local address 127.19.65.131:46781)
0619 10:22:00.769900 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.769902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.769911 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:00.769966 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.769967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.769975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.770027 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:00.770205 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.770623 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.771605 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.771967 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.772046 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.772067 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.772219 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.772252 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:22:00.772346 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.769742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:46781 (local address 127.19.65.129:44473)
0619 10:22:00.769850 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.769861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.769918 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.770045 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.770046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.770049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:00.770116 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.770118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.770696 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.770697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.771526 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.771528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.772067 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.772068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.772138 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.772141 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.772210 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.772248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":73,"thread_start_us":42,"threads_started":1}
I20260619 10:22:00.772718 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.769547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.19.65.129:60825)
0619 10:22:00.769665 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:00.769683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:00.769744 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:00.769968 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:00.769969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:00.769971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:00.770040 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.770041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.770655 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.770656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.771523 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:00.771525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.772047 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.772060 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:00.772126 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.772129 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:00.772194 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:22:00.772233 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260619 10:22:00.772696 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d1bb9027d94763a506520ded135ae9" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" is_pre_election: true
I20260619 10:22:00.772845 27642 raft_consensus.cc:2454] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 0.
I20260619 10:22:00.773072 27296 leader_election.cc:302] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d107fb1b4cfd4c96b9b8189a1075b5f4, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:00.773077 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d1bb9027d94763a506520ded135ae9" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5" is_pre_election: true
I20260619 10:22:00.773182 27722 raft_consensus.cc:2781] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:00.773234 27502 raft_consensus.cc:2454] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 0.
I20260619 10:22:00.773237 27722 raft_consensus.cc:482] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:00.773280 27722 raft_consensus.cc:3037] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:00.773948 27722 raft_consensus.cc:504] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.774098 27722 leader_election.cc:288] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:00.774297 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d1bb9027d94763a506520ded135ae9" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5"
I20260619 10:22:00.774313 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d1bb9027d94763a506520ded135ae9" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
I20260619 10:22:00.774389 27502 raft_consensus.cc:3037] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:00.774384 27642 raft_consensus.cc:3037] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:00.775082 27642 raft_consensus.cc:2454] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 1.
I20260619 10:22:00.775175 27502 raft_consensus.cc:2454] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 1.
I20260619 10:22:00.775233 27296 leader_election.cc:302] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d107fb1b4cfd4c96b9b8189a1075b5f4, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:00.775341 27722 raft_consensus.cc:2781] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:00.775440 27722 raft_consensus.cc:686] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 LEADER]: Becoming Leader. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Running, Role: LEADER
I20260619 10:22:00.775533 27722 consensus_queue.cc:237] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.776252 27204 catalog_manager.cc:5495] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 reported cstate change: term changed from 0 to 1, leader changed from <none> to e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "e26d0c143aea498ba7ca09e22c2bd066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } health_report { overall_health: UNKNOWN } } }
I20260619 10:22:00.788412 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:00.786192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.19.65.129:46111)
0619 10:22:00.786246 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:00.786248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:00.786274 (+ 26us) server_negotiation.cc:407] Connection header received
0619 10:22:00.786325 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:00.786326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:00.786339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:00.786398 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:00.786510 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.786943 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.787740 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:00.787914 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:00.787966 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:00.788118 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:00.788211 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:00.788271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:00.788333 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:22:00.788348 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:22:00.790050 27642 raft_consensus.cc:1261] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Refusing update from remote peer e26d0c143aea498ba7ca09e22c2bd066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:00.790088 27502 raft_consensus.cc:1261] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Refusing update from remote peer e26d0c143aea498ba7ca09e22c2bd066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:00.790356 27727 consensus_queue.cc:1035] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:00.790467 27722 consensus_queue.cc:1035] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:00.793484 27732 mvcc.cc:205] Tried to move back new op lower bound from 7298517077153521664 to 7298517077097099264. Current Snapshot: MvccSnapshot[applied={T|T < 7298517077153521664}]
I20260619 10:22:00.795215 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:00.795324 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:00.796027 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Sending DeleteTablet for 3 replicas of tablet 47d1bb9027d94763a506520ded135ae9
I20260619 10:22:00.796334 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d1bb9027d94763a506520ded135ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:00 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:42740
I20260619 10:22:00.796336 27482 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d1bb9027d94763a506520ded135ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:00 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43602
I20260619 10:22:00.796553 27740 tablet_replica.cc:330] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: stopping tablet replica
I20260619 10:22:00.796551 27622 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d1bb9027d94763a506520ded135ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:00 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:44038
I20260619 10:22:00.796679 27740 raft_consensus.cc:2229] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:00.796878 27739 tablet_replica.cc:330] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: stopping tablet replica
I20260619 10:22:00.796854 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:22:00.797006 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:00.797008 27739 raft_consensus.cc:2229] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:22:00.797039 27740 raft_consensus.cc:2258] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:00.797091 27739 raft_consensus.cc:2258] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:00.797592 27740 ts_tablet_manager.cc:1905] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:00.798244 27342 tablet_service.cc:1467] Processing CreateTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 (DEFAULT_TABLE table=test-table [id=e6801be8161d453abc12557b77f54c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.798321 27741 tablet_replica.cc:330] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: stopping tablet replica
I20260619 10:22:00.798382 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9cad1f2c9841bc8893752b8bd1db26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.798408 27741 raft_consensus.cc:2229] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:00.798501 27741 raft_consensus.cc:2258] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:00.798946 27741 ts_tablet_manager.cc:1905] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:00.799424 27740 ts_tablet_manager.cc:1918] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:00.799484 27740 log.cc:1196] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/47d1bb9027d94763a506520ded135ae9
I20260619 10:22:00.799726 27740 ts_tablet_manager.cc:1939] T 47d1bb9027d94763a506520ded135ae9 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting consensus metadata
I20260619 10:22:00.799799 27716 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Bootstrap starting.
I20260619 10:22:00.800040 27622 tablet_service.cc:1467] Processing CreateTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 (DEFAULT_TABLE table=test-table [id=e6801be8161d453abc12557b77f54c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.800194 27716 tablet_bootstrap.cc:654] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.800215 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9cad1f2c9841bc8893752b8bd1db26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.800836 27716 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: No bootstrap required, opened a new log
I20260619 10:22:00.800881 27716 ts_tablet_manager.cc:1397] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:00.801000 27741 ts_tablet_manager.cc:1918] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:00.801055 27741 log.cc:1196] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/47d1bb9027d94763a506520ded135ae9
I20260619 10:22:00.801040 27716 raft_consensus.cc:348] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.801110 27716 raft_consensus.cc:374] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.801142 27716 raft_consensus.cc:729] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.801187 27716 consensus_queue.cc:260] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.801234 27717 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Bootstrap starting.
I20260619 10:22:00.801277 27716 ts_tablet_manager.cc:1428] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:00.801644 27717 tablet_bootstrap.cc:654] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.801622 27192 catalog_manager.cc:4841] TS 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685): tablet 47d1bb9027d94763a506520ded135ae9 (table test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]) successfully deleted
I20260619 10:22:00.801764 27482 tablet_service.cc:1467] Processing CreateTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 (DEFAULT_TABLE table=test-table [id=e6801be8161d453abc12557b77f54c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:00.801853 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9cad1f2c9841bc8893752b8bd1db26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:00.802383 27741 ts_tablet_manager.cc:1939] T 47d1bb9027d94763a506520ded135ae9 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting consensus metadata
I20260619 10:22:00.802778 27192 catalog_manager.cc:4841] TS d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781): tablet 47d1bb9027d94763a506520ded135ae9 (table test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]) successfully deleted
I20260619 10:22:00.802798 27718 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Bootstrap starting.
I20260619 10:22:00.803059 27739 ts_tablet_manager.cc:1905] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:00.803179 27718 tablet_bootstrap.cc:654] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:00.803249 27717 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: No bootstrap required, opened a new log
I20260619 10:22:00.803308 27717 ts_tablet_manager.cc:1397] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:22:00.803436 27717 raft_consensus.cc:348] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.803495 27717 raft_consensus.cc:374] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.803524 27717 raft_consensus.cc:729] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.803562 27717 consensus_queue.cc:260] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.803650 27717 ts_tablet_manager.cc:1428] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:00.804816 27739 ts_tablet_manager.cc:1918] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:00.804872 27739 log.cc:1196] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/47d1bb9027d94763a506520ded135ae9
I20260619 10:22:00.805096 27739 ts_tablet_manager.cc:1939] T 47d1bb9027d94763a506520ded135ae9 P e26d0c143aea498ba7ca09e22c2bd066: Deleting consensus metadata
I20260619 10:22:00.805446 27189 catalog_manager.cc:4841] TS e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111): tablet 47d1bb9027d94763a506520ded135ae9 (table test-table [id=254ccaaaf1ba4b3f80e5ae5762fffc63]) successfully deleted
I20260619 10:22:00.807639 27718 tablet_bootstrap.cc:492] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: No bootstrap required, opened a new log
I20260619 10:22:00.807679 27718 ts_tablet_manager.cc:1397] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 10:22:00.807844 27718 raft_consensus.cc:348] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.807896 27718 raft_consensus.cc:374] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:00.807909 27718 raft_consensus.cc:729] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Initialized, Role: FOLLOWER
I20260619 10:22:00.807958 27718 consensus_queue.cc:260] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:00.808050 27718 ts_tablet_manager.cc:1428] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260619 10:22:00.808058 27689 tablet.cc:2369] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:22:01.008677 27723 raft_consensus.cc:482] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:01.008785 27723 raft_consensus.cc:504] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.009146 27723 leader_election.cc:288] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.012292 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.009456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:53569 (local address 127.19.65.131:46781)
0619 10:22:01.009502 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.009504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.009668 (+ 164us) server_negotiation.cc:407] Connection header received
0619 10:22:01.009735 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.009736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.009745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.009802 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:01.009962 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.010474 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.011495 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.011932 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.011990 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.011997 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.012173 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.012211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:22:01.012319 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.009414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60911 (local address 127.19.65.129:46111)
0619 10:22:01.009485 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.009488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.009611 (+ 123us) server_negotiation.cc:407] Connection header received
0619 10:22:01.009660 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.009662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.009675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.009727 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:01.009910 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.010418 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.011526 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.011953 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.012045 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.012053 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.012201 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.012249 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:22:01.012344 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.009360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46111 (local address 127.19.65.130:60911)
0619 10:22:01.009535 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.009545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.009600 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.009741 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.009742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.009745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:01.009832 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.009835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.010491 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.010492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.011422 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.011424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.012051 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.012052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.012110 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.012114 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.012185 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.012229 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":26,"thread_start_us":65,"threads_started":1}
I20260619 10:22:01.012353 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.009362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:46781 (local address 127.19.65.130:53569)
0619 10:22:01.009556 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.009565 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.009659 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.009819 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.009820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.009821 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:01.009890 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.009891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.010564 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.010566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.011414 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.011417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.012014 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.012015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.012098 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.012107 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.012149 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.012210 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":86,"thread_start_us":37,"threads_started":1}
I20260619 10:22:01.012598 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9cad1f2c9841bc8893752b8bd1db26" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066" is_pre_election: true
I20260619 10:22:01.012751 27362 raft_consensus.cc:2454] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 0.
I20260619 10:22:01.012849 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9cad1f2c9841bc8893752b8bd1db26" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" is_pre_election: true
I20260619 10:22:01.012939 27642 raft_consensus.cc:2454] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 0.
I20260619 10:22:01.012971 27433 leader_election.cc:302] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:01.013104 27723 raft_consensus.cc:2781] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:01.013146 27723 raft_consensus.cc:482] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:01.013171 27723 raft_consensus.cc:3037] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.013844 27723 raft_consensus.cc:504] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.013996 27723 leader_election.cc:288] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 election: Requested vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.014137 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9cad1f2c9841bc8893752b8bd1db26" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066"
I20260619 10:22:01.014163 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9cad1f2c9841bc8893752b8bd1db26" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
I20260619 10:22:01.014230 27362 raft_consensus.cc:3037] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.014253 27642 raft_consensus.cc:3037] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.014818 27642 raft_consensus.cc:2454] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 1.
I20260619 10:22:01.014945 27362 raft_consensus.cc:2454] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 1.
I20260619 10:22:01.015079 27436 leader_election.cc:302] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, d107fb1b4cfd4c96b9b8189a1075b5f4; no voters:
I20260619 10:22:01.015154 27723 raft_consensus.cc:2781] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:01.015285 27723 raft_consensus.cc:686] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Running, Role: LEADER
I20260619 10:22:01.015413 27723 consensus_queue.cc:237] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.016095 27203 catalog_manager.cc:5495] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864519, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.190@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:01.047405 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.040128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.19.65.190:35213)
0619 10:22:01.040196 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.040198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.040322 (+ 124us) server_negotiation.cc:407] Connection header received
0619 10:22:01.040360 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.040361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.040369 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.040424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:01.040985 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.041447 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.042158 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.042326 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.042371 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.044512 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:01.044513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:01.044513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:01.044519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:01.046198 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:01.046614 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:01.046615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:01.046615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:01.046672 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:01.046892 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:01.046893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:01.046893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:01.047018 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:01.047092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.047203 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.047334 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:22:01.047535 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.040067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35213 (local address 127.0.0.1:34618)
0619 10:22:01.040240 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.040252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.040319 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.040541 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.040543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.040863 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:01.040912 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.040915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.041534 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.041535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.042085 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.042086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.042388 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.042388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.042429 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.042998 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:01.043002 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:01.044418 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:01.046324 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:01.046330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:01.046332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:01.046550 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:01.046740 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:01.046740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:01.046740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:01.046795 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:01.047089 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:01.047094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:01.047133 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.047178 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.047393 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":96,"thread_start_us":124,"threads_started":1}
I20260619 10:22:01.049396 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:22:01.049587 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:01.051107 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 085259ae1020442cbad6bd0cf0ec6710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca7b10e78fa429eb67cb5aee91d1466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:22:01.051146 27482 tablet_service.cc:1467] Processing CreateTablet for tablet 085259ae1020442cbad6bd0cf0ec6710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca7b10e78fa429eb67cb5aee91d1466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:22:01.051292 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085259ae1020442cbad6bd0cf0ec6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.051295 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085259ae1020442cbad6bd0cf0ec6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.051437 27622 tablet_service.cc:1467] Processing CreateTablet for tablet 085259ae1020442cbad6bd0cf0ec6710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca7b10e78fa429eb67cb5aee91d1466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:22:01.051576 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085259ae1020442cbad6bd0cf0ec6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.052413 27718 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: Bootstrap starting.
I20260619 10:22:01.052769 27718 tablet_bootstrap.cc:654] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.052906 27716 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: Bootstrap starting.
I20260619 10:22:01.053097 27717 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: Bootstrap starting.
I20260619 10:22:01.053234 27716 tablet_bootstrap.cc:654] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.053542 27717 tablet_bootstrap.cc:654] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.053776 27718 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: No bootstrap required, opened a new log
I20260619 10:22:01.053838 27718 ts_tablet_manager.cc:1397] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:01.054011 27718 raft_consensus.cc:348] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054117 27718 raft_consensus.cc:374] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.054144 27718 raft_consensus.cc:729] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.054198 27718 consensus_queue.cc:260] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054272 27716 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: No bootstrap required, opened a new log
I20260619 10:22:01.054293 27718 ts_tablet_manager.cc:1428] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:01.054324 27716 ts_tablet_manager.cc:1397] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:01.054380 27723 raft_consensus.cc:482] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:01.054397 27717 tablet_bootstrap.cc:492] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: No bootstrap required, opened a new log
I20260619 10:22:01.054406 27723 raft_consensus.cc:504] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054437 27717 ts_tablet_manager.cc:1397] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:01.054443 27716 raft_consensus.cc:348] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054502 27723 leader_election.cc:288] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.054519 27716 raft_consensus.cc:374] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.054544 27716 raft_consensus.cc:729] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.054579 27723 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.054605 27716 consensus_queue.cc:260] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054594 27717 raft_consensus.cc:348] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054652 27723 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.054670 27717 raft_consensus.cc:374] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.054695 27717 raft_consensus.cc:729] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.054709 27716 ts_tablet_manager.cc:1428] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:01.054742 27717 consensus_queue.cc:260] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.054750 27722 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.054840 27722 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.054873 27717 ts_tablet_manager.cc:1428] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:01.054898 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085259ae1020442cbad6bd0cf0ec6710" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066" is_pre_election: true
I20260619 10:22:01.054953 27362 raft_consensus.cc:2454] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 0.
I20260619 10:22:01.055029 27724 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.055106 27433 leader_election.cc:302] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:01.055104 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085259ae1020442cbad6bd0cf0ec6710" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" is_pre_election: true
I20260619 10:22:01.055178 27642 raft_consensus.cc:2454] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 0.
I20260619 10:22:01.055258 27747 raft_consensus.cc:2781] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:01.055419 27747 raft_consensus.cc:482] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:01.055451 27747 raft_consensus.cc:3037] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.056126 27747 raft_consensus.cc:504] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.056269 27747 leader_election.cc:288] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 election: Requested vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.055111 27724 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.056553 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085259ae1020442cbad6bd0cf0ec6710" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
I20260619 10:22:01.056627 27642 raft_consensus.cc:3037] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.056704 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085259ae1020442cbad6bd0cf0ec6710" candidate_uuid: "2b3626e99ac64a1684327ecc1d7affc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066"
I20260619 10:22:01.056782 27362 raft_consensus.cc:3037] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.057199 27642 raft_consensus.cc:2454] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 1.
I20260619 10:22:01.057232 27362 raft_consensus.cc:2454] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3626e99ac64a1684327ecc1d7affc5 in term 1.
I20260619 10:22:01.057358 27436 leader_election.cc:302] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, d107fb1b4cfd4c96b9b8189a1075b5f4; no voters:
I20260619 10:22:01.057441 27747 raft_consensus.cc:2781] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:01.057502 27747 raft_consensus.cc:686] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Running, Role: LEADER
I20260619 10:22:01.057549 27747 consensus_queue.cc:237] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.057823 27747 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: TxnStatusTablet state changed. Reason: New leader 2b3626e99ac64a1684327ecc1d7affc5. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.057894 27747 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:22:01.058070 27759 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:22:01.058066 27203 catalog_manager.cc:5495] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } health_report { overall_health: UNKNOWN } } }
I20260619 10:22:01.058172 27759 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:22:01.064517 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.061816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.0.0.1:43612)
0619 10:22:01.061883 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.061895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.061938 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.062105 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.062106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.062107 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:22:01.062145 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.062147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.063006 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.063007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.063635 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.063637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.064085 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.064086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.064124 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.064137 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.064390 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.064396 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.064433 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.064467 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:22:01.064546 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.061902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.19.65.130:46685)
0619 10:22:01.061942 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.061944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.061953 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:01.062006 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.062007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.062024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.062107 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.062186 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.062924 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.063710 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.064009 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.064083 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.064212 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.064325 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.064402 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.064439 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.064461 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:22:01.065630 27642 raft_consensus.cc:1261] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Refusing update from remote peer 2b3626e99ac64a1684327ecc1d7affc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.065706 27362 raft_consensus.cc:1261] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Refusing update from remote peer 2b3626e99ac64a1684327ecc1d7affc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.065915 27747 consensus_queue.cc:1035] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.066013 27723 consensus_queue.cc:1035] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.066519 27724 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: TxnStatusTablet state changed. Reason: New leader 2b3626e99ac64a1684327ecc1d7affc5. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.066598 27724 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.066579 27722 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: TxnStatusTablet state changed. Reason: New leader 2b3626e99ac64a1684327ecc1d7affc5. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.066671 27722 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.067229 27747 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.067337 27747 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:22:01.067663 27724 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.067741 27724 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P d107fb1b4cfd4c96b9b8189a1075b5f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.071269 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.068934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.19.65.130:46685)
0619 10:22:01.068966 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.068968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.068974 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:22:01.068989 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.068990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.068997 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.069041 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.069175 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.069657 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.070695 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.070878 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.070937 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.071088 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.071111 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.071144 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.071211 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.071224 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:22:01.071858 27747 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.071962 27747 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P 2b3626e99ac64a1684327ecc1d7affc5: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:22:01.072535 27722 tablet_replica.cc:439] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3626e99ac64a1684327ecc1d7affc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } } }
I20260619 10:22:01.072613 27722 tablet_replica.cc:442] T 085259ae1020442cbad6bd0cf0ec6710 P e26d0c143aea498ba7ca09e22c2bd066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:22:01.076601 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.073826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:53189 (local address 127.19.65.130:46685)
0619 10:22:01.073861 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.073863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.074012 (+ 149us) server_negotiation.cc:407] Connection header received
0619 10:22:01.074039 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.074039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.074048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.074105 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.074251 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.074953 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.075766 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.076069 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.076137 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.076307 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.076337 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.076389 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.076496 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.076512 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:22:01.076618 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.073782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.19.65.130:53189)
0619 10:22:01.073939 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.073949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.074008 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.074112 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.074113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.074114 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:22:01.074195 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.074195 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.075028 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.075029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.075695 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.075697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.076149 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.076150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.076221 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.076233 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.076429 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.076432 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.076485 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.076520 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260619 10:22:01.078912 27362 raft_consensus.cc:1261] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Refusing update from remote peer 2b3626e99ac64a1684327ecc1d7affc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.078943 27642 raft_consensus.cc:1261] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Refusing update from remote peer 2b3626e99ac64a1684327ecc1d7affc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.079110 27764 consensus_queue.cc:1035] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.079195 27723 consensus_queue.cc:1035] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.081241 27730 mvcc.cc:205] Tried to move back new op lower bound from 7298517078337986560 to 7298517078079778816. Current Snapshot: MvccSnapshot[applied={T|T < 7298517078337986560}]
I20260619 10:22:01.093331 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.093436 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.094018 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Sending DeleteTablet for 3 replicas of tablet ad9cad1f2c9841bc8893752b8bd1db26
I20260619 10:22:01.094319 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:42740
I20260619 10:22:01.094339 27622 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:44038
I20260619 10:22:01.094424 27739 tablet_replica.cc:330] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: stopping tablet replica
I20260619 10:22:01.094444 27741 tablet_replica.cc:330] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: stopping tablet replica
I20260619 10:22:01.094483 27741 raft_consensus.cc:2229] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.094482 27739 raft_consensus.cc:2229] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.094518 27741 raft_consensus.cc:2258] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.094525 27739 raft_consensus.cc:2258] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.094843 27741 ts_tablet_manager.cc:1905] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.095481 27739 ts_tablet_manager.cc:1905] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.096916 27741 ts_tablet_manager.cc:1918] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:22:01.096976 27741 log.cc:1196] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/ad9cad1f2c9841bc8893752b8bd1db26
I20260619 10:22:01.096982 27739 ts_tablet_manager.cc:1918] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:22:01.097034 27739 log.cc:1196] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/ad9cad1f2c9841bc8893752b8bd1db26
I20260619 10:22:01.097232 27741 ts_tablet_manager.cc:1939] T ad9cad1f2c9841bc8893752b8bd1db26 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting consensus metadata
I20260619 10:22:01.097250 27739 ts_tablet_manager.cc:1939] T ad9cad1f2c9841bc8893752b8bd1db26 P e26d0c143aea498ba7ca09e22c2bd066: Deleting consensus metadata
I20260619 10:22:01.097549 27192 catalog_manager.cc:4841] TS d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781): tablet ad9cad1f2c9841bc8893752b8bd1db26 (table test-table [id=e6801be8161d453abc12557b77f54c8f]) successfully deleted
I20260619 10:22:01.097661 27189 catalog_manager.cc:4841] TS e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111): tablet ad9cad1f2c9841bc8893752b8bd1db26 (table test-table [id=e6801be8161d453abc12557b77f54c8f]) successfully deleted
I20260619 10:22:01.097672 27482 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9cad1f2c9841bc8893752b8bd1db26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43602
I20260619 10:22:01.097759 27740 tablet_replica.cc:330] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: stopping tablet replica
I20260619 10:22:01.097811 27740 raft_consensus.cc:2229] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:22:01.097908 27740 raft_consensus.cc:2258] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.098202 27740 ts_tablet_manager.cc:1905] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.100113 27740 ts_tablet_manager.cc:1918] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:22:01.100186 27740 log.cc:1196] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/ad9cad1f2c9841bc8893752b8bd1db26
I20260619 10:22:01.100412 27740 ts_tablet_manager.cc:1939] T ad9cad1f2c9841bc8893752b8bd1db26 P 2b3626e99ac64a1684327ecc1d7affc5: Deleting consensus metadata
I20260619 10:22:01.100747 27192 catalog_manager.cc:4841] TS 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685): tablet ad9cad1f2c9841bc8893752b8bd1db26 (table test-table [id=e6801be8161d453abc12557b77f54c8f]) successfully deleted
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:22:01.120029 27255 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:01.113952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.19.65.190:35213)
0619 10:22:01.114022 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.114025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.114169 (+ 144us) server_negotiation.cc:407] Connection header received
0619 10:22:01.114242 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.114242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.114267 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.114317 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:01.115017 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.115597 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.116720 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.116908 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.116956 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.119964 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34628: BlockingRecv error: recv got EOF from 127.0.0.1:34628 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM
I20260619 10:22:01.128381 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.120791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.19.65.190:35213)
0619 10:22:01.120828 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.120830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.120903 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:22:01.120942 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.120942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.120951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.121008 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:01.121402 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.122101 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.122804 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.123105 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.123173 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.125456 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:01.125457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:01.125458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:01.125465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:01.127145 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:01.127563 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:01.127564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:01.127564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:01.127631 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:01.127856 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:01.127857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:01.127858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:01.127974 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:01.128082 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.128197 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.128278 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:22:01.129979 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:22:01.130184 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:01.131601 27342 tablet_service.cc:1467] Processing CreateTablet for tablet b56e0272cecf48488e48c6f3a45590ca (DEFAULT_TABLE table=test-table [id=57913e8e18ab42a09789d9c767bd1f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.131668 27622 tablet_service.cc:1467] Processing CreateTablet for tablet b56e0272cecf48488e48c6f3a45590ca (DEFAULT_TABLE table=test-table [id=57913e8e18ab42a09789d9c767bd1f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.131747 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e0272cecf48488e48c6f3a45590ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.131778 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e0272cecf48488e48c6f3a45590ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.133114 27717 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Bootstrap starting.
I20260619 10:22:01.133167 27482 tablet_service.cc:1467] Processing CreateTablet for tablet b56e0272cecf48488e48c6f3a45590ca (DEFAULT_TABLE table=test-table [id=57913e8e18ab42a09789d9c767bd1f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.133307 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e0272cecf48488e48c6f3a45590ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.133569 27717 tablet_bootstrap.cc:654] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.133917 27716 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Bootstrap starting.
I20260619 10:22:01.134231 27716 tablet_bootstrap.cc:654] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.134591 27718 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Bootstrap starting.
I20260619 10:22:01.134876 27716 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: No bootstrap required, opened a new log
I20260619 10:22:01.134929 27716 ts_tablet_manager.cc:1397] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:01.135008 27718 tablet_bootstrap.cc:654] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.135030 27716 raft_consensus.cc:348] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.135097 27716 raft_consensus.cc:374] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.135121 27716 raft_consensus.cc:729] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.135160 27716 consensus_queue.cc:260] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.135248 27716 ts_tablet_manager.cc:1428] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:01.135614 27717 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: No bootstrap required, opened a new log
I20260619 10:22:01.135677 27717 ts_tablet_manager.cc:1397] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 10:22:01.135831 27717 raft_consensus.cc:348] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.135900 27717 raft_consensus.cc:374] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.135921 27717 raft_consensus.cc:729] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.135960 27717 consensus_queue.cc:260] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.136044 27717 ts_tablet_manager.cc:1428] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:01.136443 27718 tablet_bootstrap.cc:492] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: No bootstrap required, opened a new log
I20260619 10:22:01.136515 27718 ts_tablet_manager.cc:1397] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:22:01.136686 27718 raft_consensus.cc:348] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.136746 27718 raft_consensus.cc:374] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.136762 27718 raft_consensus.cc:729] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.136821 27718 consensus_queue.cc:260] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.136945 27718 ts_tablet_manager.cc:1428] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:01.223789 27722 raft_consensus.cc:482] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:01.223894 27722 raft_consensus.cc:504] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.224090 27722 leader_election.cc:288] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.224375 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e0272cecf48488e48c6f3a45590ca" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5" is_pre_election: true
I20260619 10:22:01.224409 27641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e0272cecf48488e48c6f3a45590ca" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" is_pre_election: true
I20260619 10:22:01.224485 27502 raft_consensus.cc:2454] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 0.
I20260619 10:22:01.224509 27641 raft_consensus.cc:2454] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 0.
I20260619 10:22:01.224684 27296 leader_election.cc:302] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d107fb1b4cfd4c96b9b8189a1075b5f4, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:01.224805 27722 raft_consensus.cc:2781] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:01.224864 27722 raft_consensus.cc:482] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:01.224885 27722 raft_consensus.cc:3037] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.225772 27722 raft_consensus.cc:504] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.225898 27722 leader_election.cc:288] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685), d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781)
I20260619 10:22:01.226126 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e0272cecf48488e48c6f3a45590ca" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5"
I20260619 10:22:01.226138 27641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e0272cecf48488e48c6f3a45590ca" candidate_uuid: "e26d0c143aea498ba7ca09e22c2bd066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4"
I20260619 10:22:01.226203 27641 raft_consensus.cc:3037] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.226202 27502 raft_consensus.cc:3037] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.227005 27502 raft_consensus.cc:2454] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 1.
I20260619 10:22:01.227173 27296 leader_election.cc:302] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:01.227268 27641 raft_consensus.cc:2454] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26d0c143aea498ba7ca09e22c2bd066 in term 1.
I20260619 10:22:01.227284 27722 raft_consensus.cc:2781] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:01.227349 27722 raft_consensus.cc:686] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 1 LEADER]: Becoming Leader. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Running, Role: LEADER
I20260619 10:22:01.227403 27722 consensus_queue.cc:237] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.227933 27204 catalog_manager.cc:5495] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 reported cstate change: term changed from 0 to 1, leader changed from <none> to e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "e26d0c143aea498ba7ca09e22c2bd066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } health_report { overall_health: UNKNOWN } } }
I20260619 10:22:01.255262 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.252404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.19.65.129:46111)
0619 10:22:01.252445 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.252446 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.252481 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:22:01.252502 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.252502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.252511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.252547 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.252688 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.253318 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.254434 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.254733 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.254801 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.255012 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.255050 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.255100 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.255169 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.255192 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:22:01.255893 27641 raft_consensus.cc:1261] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Refusing update from remote peer e26d0c143aea498ba7ca09e22c2bd066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.255904 27502 raft_consensus.cc:1261] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Refusing update from remote peer e26d0c143aea498ba7ca09e22c2bd066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.256111 27727 consensus_queue.cc:1035] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.256223 27727 consensus_queue.cc:1035] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.257723 27731 mvcc.cc:205] Tried to move back new op lower bound from 7298517079062753280 to 7298517078947651584. Current Snapshot: MvccSnapshot[applied={T|T < 7298517079062753280 or (T in {7298517079062753280})}]
I20260619 10:22:01.258757 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.258873 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.259446 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Sending DeleteTablet for 3 replicas of tablet b56e0272cecf48488e48c6f3a45590ca
I20260619 10:22:01.259668 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet b56e0272cecf48488e48c6f3a45590ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:42740
I20260619 10:22:01.259668 27482 tablet_service.cc:1514] Processing DeleteTablet for tablet b56e0272cecf48488e48c6f3a45590ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43602
I20260619 10:22:01.259717 27622 tablet_service.cc:1514] Processing DeleteTablet for tablet b56e0272cecf48488e48c6f3a45590ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:44038
I20260619 10:22:01.259752 27739 tablet_replica.cc:330] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: stopping tablet replica
I20260619 10:22:01.259790 27739 raft_consensus.cc:2229] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:22:01.259794 27740 tablet_replica.cc:330] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: stopping tablet replica
I20260619 10:22:01.259833 27740 raft_consensus.cc:2229] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.259841 27739 raft_consensus.cc:2258] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.259881 27740 raft_consensus.cc:2258] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.259943 27741 tablet_replica.cc:330] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: stopping tablet replica
I20260619 10:22:01.259977 27741 raft_consensus.cc:2229] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.260013 27741 raft_consensus.cc:2258] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.260172 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:22:01.260293 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:01.260391 27740 ts_tablet_manager.cc:1905] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.260591 27739 ts_tablet_manager.cc:1905] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.261861 27741 ts_tablet_manager.cc:1905] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.261919 27740 ts_tablet_manager.cc:1918] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:01.261965 27740 log.cc:1196] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/b56e0272cecf48488e48c6f3a45590ca
I20260619 10:22:01.262063 27739 ts_tablet_manager.cc:1918] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:01.262109 27739 log.cc:1196] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/b56e0272cecf48488e48c6f3a45590ca
I20260619 10:22:01.262181 27740 ts_tablet_manager.cc:1939] T b56e0272cecf48488e48c6f3a45590ca P 2b3626e99ac64a1684327ecc1d7affc5: Deleting consensus metadata
I20260619 10:22:01.262302 27739 ts_tablet_manager.cc:1939] T b56e0272cecf48488e48c6f3a45590ca P e26d0c143aea498ba7ca09e22c2bd066: Deleting consensus metadata
I20260619 10:22:01.262487 27192 catalog_manager.cc:4841] TS 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685): tablet b56e0272cecf48488e48c6f3a45590ca (table test-table [id=57913e8e18ab42a09789d9c767bd1f9d]) successfully deleted
I20260619 10:22:01.262851 27189 catalog_manager.cc:4841] TS e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111): tablet b56e0272cecf48488e48c6f3a45590ca (table test-table [id=57913e8e18ab42a09789d9c767bd1f9d]) successfully deleted
I20260619 10:22:01.263440 27741 ts_tablet_manager.cc:1918] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:01.263490 27741 log.cc:1196] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/b56e0272cecf48488e48c6f3a45590ca
I20260619 10:22:01.263623 27622 tablet_service.cc:1467] Processing CreateTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 (DEFAULT_TABLE table=test-table [id=2b683f1372584b68a99306eaf7cf124b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.263710 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f43e21a0d74bf1a51ecd263c48a236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.263722 27741 ts_tablet_manager.cc:1939] T b56e0272cecf48488e48c6f3a45590ca P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting consensus metadata
I20260619 10:22:01.264590 27717 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Bootstrap starting.
I20260619 10:22:01.264633 27192 catalog_manager.cc:4841] TS d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781): tablet b56e0272cecf48488e48c6f3a45590ca (table test-table [id=57913e8e18ab42a09789d9c767bd1f9d]) successfully deleted
I20260619 10:22:01.264784 27482 tablet_service.cc:1467] Processing CreateTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 (DEFAULT_TABLE table=test-table [id=2b683f1372584b68a99306eaf7cf124b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.264905 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f43e21a0d74bf1a51ecd263c48a236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.265003 27717 tablet_bootstrap.cc:654] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.265641 27717 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: No bootstrap required, opened a new log
I20260619 10:22:01.265695 27717 ts_tablet_manager.cc:1397] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:01.265828 27717 raft_consensus.cc:348] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.265918 27717 raft_consensus.cc:374] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.265946 27717 raft_consensus.cc:729] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.265985 27717 consensus_queue.cc:260] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.266024 27718 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: Bootstrap starting.
I20260619 10:22:01.266074 27717 ts_tablet_manager.cc:1428] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:01.266285 27718 tablet_bootstrap.cc:654] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.266295 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 (DEFAULT_TABLE table=test-table [id=2b683f1372584b68a99306eaf7cf124b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:01.266383 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f43e21a0d74bf1a51ecd263c48a236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:01.267280 27716 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: Bootstrap starting.
I20260619 10:22:01.267580 27716 tablet_bootstrap.cc:654] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:01.267724 27718 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: No bootstrap required, opened a new log
I20260619 10:22:01.267786 27718 ts_tablet_manager.cc:1397] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260619 10:22:01.267884 27718 raft_consensus.cc:348] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.267940 27718 raft_consensus.cc:374] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.267966 27718 raft_consensus.cc:729] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3626e99ac64a1684327ecc1d7affc5, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.268010 27718 consensus_queue.cc:260] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.268106 27718 ts_tablet_manager.cc:1428] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:01.268267 27716 tablet_bootstrap.cc:492] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: No bootstrap required, opened a new log
I20260619 10:22:01.268319 27716 ts_tablet_manager.cc:1397] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:01.268412 27716 raft_consensus.cc:348] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.268474 27716 raft_consensus.cc:374] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:01.268493 27716 raft_consensus.cc:729] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26d0c143aea498ba7ca09e22c2bd066, State: Initialized, Role: FOLLOWER
I20260619 10:22:01.268528 27716 consensus_queue.cc:260] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.268608 27716 ts_tablet_manager.cc:1428] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:01.484880 27724 raft_consensus.cc:482] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:01.484989 27724 raft_consensus.cc:504] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.485347 27724 leader_election.cc:288] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685)
I20260619 10:22:01.488523 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.485675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:53293 (local address 127.19.65.129:46111)
0619 10:22:01.485728 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.485730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.485786 (+ 56us) server_negotiation.cc:407] Connection header received
0619 10:22:01.485918 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.485920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.485933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.485990 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:01.486187 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.486607 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.487802 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.488138 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.488193 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.488201 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.488416 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.488454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:22:01.488526 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.485656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:47671 (local address 127.19.65.130:46685)
0619 10:22:01.485701 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.485704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.485780 (+ 76us) server_negotiation.cc:407] Connection header received
0619 10:22:01.485848 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.485850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.485861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.485926 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:01.486091 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.486614 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.487764 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.488157 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.488262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.488271 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.488414 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.488451 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:22:01.488566 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.485539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.19.65.131:47671)
0619 10:22:01.485697 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.485709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.485788 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.485932 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.485933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.485936 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:01.486015 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.486017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.486738 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.486739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.487671 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.487673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.488279 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.488280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.488331 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.488335 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.488409 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.488448 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":16,"thread_start_us":111,"threads_started":1}
I20260619 10:22:01.488571 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.485550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46111 (local address 127.19.65.131:53293)
0619 10:22:01.485715 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.485727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.485814 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.486017 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.486018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.486020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:01.486112 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.486114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.486689 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.486690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.487689 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.487692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.488249 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.488250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.488325 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.488334 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.488411 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.488453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":63,"thread_start_us":85,"threads_started":1}
I20260619 10:22:01.488793 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f43e21a0d74bf1a51ecd263c48a236" candidate_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5" is_pre_election: true
I20260619 10:22:01.488889 27502 raft_consensus.cc:2454] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d107fb1b4cfd4c96b9b8189a1075b5f4 in term 0.
I20260619 10:22:01.488870 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f43e21a0d74bf1a51ecd263c48a236" candidate_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066" is_pre_election: true
I20260619 10:22:01.488962 27362 raft_consensus.cc:2454] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d107fb1b4cfd4c96b9b8189a1075b5f4 in term 0.
I20260619 10:22:01.489074 27576 leader_election.cc:302] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3626e99ac64a1684327ecc1d7affc5, d107fb1b4cfd4c96b9b8189a1075b5f4; no voters:
I20260619 10:22:01.489224 27724 raft_consensus.cc:2781] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:01.489279 27724 raft_consensus.cc:482] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:01.489303 27724 raft_consensus.cc:3037] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.489946 27724 raft_consensus.cc:504] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.490087 27724 leader_election.cc:288] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [CANDIDATE]: Term 1 election: Requested vote from peers e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111), 2b3626e99ac64a1684327ecc1d7affc5 (127.19.65.130:46685)
I20260619 10:22:01.490281 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f43e21a0d74bf1a51ecd263c48a236" candidate_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3626e99ac64a1684327ecc1d7affc5"
I20260619 10:22:01.490280 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f43e21a0d74bf1a51ecd263c48a236" candidate_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26d0c143aea498ba7ca09e22c2bd066"
I20260619 10:22:01.490368 27362 raft_consensus.cc:3037] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.490367 27502 raft_consensus.cc:3037] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:01.491154 27502 raft_consensus.cc:2454] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d107fb1b4cfd4c96b9b8189a1075b5f4 in term 1.
I20260619 10:22:01.491151 27362 raft_consensus.cc:2454] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d107fb1b4cfd4c96b9b8189a1075b5f4 in term 1.
I20260619 10:22:01.491338 27573 leader_election.cc:302] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d107fb1b4cfd4c96b9b8189a1075b5f4, e26d0c143aea498ba7ca09e22c2bd066; no voters:
I20260619 10:22:01.491462 27724 raft_consensus.cc:2781] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:01.491555 27724 raft_consensus.cc:686] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 LEADER]: Becoming Leader. State: Replica: d107fb1b4cfd4c96b9b8189a1075b5f4, State: Running, Role: LEADER
I20260619 10:22:01.491621 27724 consensus_queue.cc:237] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } }
I20260619 10:22:01.492326 27204 catalog_manager.cc:5495] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d107fb1b4cfd4c96b9b8189a1075b5f4" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46781 } health_report { overall_health: HEALTHY } } }
I20260619 10:22:01.553797 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.551696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.19.65.131:46781)
0619 10:22:01.551808 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.551809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.551816 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:22:01.551845 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.551846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.551864 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.551932 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.552027 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.552491 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.553134 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.553309 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.553356 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.553469 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.553564 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.553611 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.553646 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.553684 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260619 10:22:01.558321 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.555975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59973 (local address 127.19.65.130:46685)
0619 10:22:01.556037 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.556039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.556101 (+ 62us) server_negotiation.cc:407] Connection header received
0619 10:22:01.556163 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.556164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.556173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.556210 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.556329 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.556786 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.557604 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.557799 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.557868 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.558064 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.558107 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.558165 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.558222 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.558241 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:22:01.558338 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.555918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.19.65.131:59973)
0619 10:22:01.556047 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.556056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.556112 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.556212 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.556212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.556214 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:22:01.556282 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.556283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.556868 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.556869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.557505 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.557507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.557900 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.557901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.557962 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.557974 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.558166 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.558169 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.558218 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.558258 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260619 10:22:01.560546 27362 raft_consensus.cc:1261] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Refusing update from remote peer d107fb1b4cfd4c96b9b8189a1075b5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.560588 27502 raft_consensus.cc:1261] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Refusing update from remote peer d107fb1b4cfd4c96b9b8189a1075b5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:01.560846 27795 consensus_queue.cc:1035] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3626e99ac64a1684327ecc1d7affc5" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.560947 27724 consensus_queue.cc:1035] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26d0c143aea498ba7ca09e22c2bd066" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:01.567708 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.565433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57103 (local address 127.19.65.131:46781)
0619 10:22:01.565493 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:01.565495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:01.565504 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:01.565531 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:01.565532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:01.565540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:01.565586 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:01.565708 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.566238 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.566984 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.567226 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.567313 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.567463 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.567498 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.567547 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:01.567619 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:22:01.567639 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:22:01.567745 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.565405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:46781 (local address 127.19.65.130:57103)
0619 10:22:01.565448 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.565458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:01.565493 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:01.565604 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:01.565605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:01.565606 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:22:01.565649 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.565650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.566314 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.566315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.566866 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:01.566867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:01.567316 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:01.567317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:01.567373 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:01.567380 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:22:01.567549 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:22:01.567551 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:22:01.567605 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:22:01.567664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 10:22:01.574401 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.574506 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:01.575088 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be480d2c51b4bed9f66781666db6b77: Sending DeleteTablet for 3 replicas of tablet 88f43e21a0d74bf1a51ecd263c48a236
I20260619 10:22:01.575332 27622 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:44038
I20260619 10:22:01.575373 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:42740
I20260619 10:22:01.575374 27482 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f43e21a0d74bf1a51ecd263c48a236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:01 UTC) from {username='oryx', principal='oryx/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43602
I20260619 10:22:01.575423 27741 tablet_replica.cc:330] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: stopping tablet replica
I20260619 10:22:01.575457 27740 tablet_replica.cc:330] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5: stopping tablet replica
I20260619 10:22:01.575472 27741 raft_consensus.cc:2229] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:22:01.575465 27739 tablet_replica.cc:330] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: stopping tablet replica
I20260619 10:22:01.575506 27740 raft_consensus.cc:2229] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.575519 27739 raft_consensus.cc:2229] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:01.575539 27740 raft_consensus.cc:2258] T 88f43e21a0d74bf1a51ecd263c48a236 P 2b3626e99ac64a1684327ecc1d7affc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.575549 27739 raft_consensus.cc:2258] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.575548 27741 raft_consensus.cc:2258] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:01.575817 27739 ts_tablet_manager.cc:1905] T 88f43e21a0d74bf1a51ecd263c48a236 P e26d0c143aea498ba7ca09e22c2bd066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.575891 27741 ts_tablet_manager.cc:1905] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:01.576159 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27277
I20260619 10:22:01.577616 27741 ts_tablet_manager.cc:1918] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:22:01.577665 27741 log.cc:1196] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/88f43e21a0d74bf1a51ecd263c48a236
I20260619 10:22:01.577884 27741 ts_tablet_manager.cc:1939] T 88f43e21a0d74bf1a51ecd263c48a236 P d107fb1b4cfd4c96b9b8189a1075b5f4: Deleting consensus metadata
I20260619 10:22:01.578867 27192 catalog_manager.cc:4841] TS d107fb1b4cfd4c96b9b8189a1075b5f4 (127.19.65.131:46781): tablet 88f43e21a0d74bf1a51ecd263c48a236 (table test-table [id=2b683f1372584b68a99306eaf7cf124b]) successfully deleted
W20260619 10:22:01.583217 27189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:46111 (error 108)
I20260619 10:22:01.583559 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27417
I20260619 10:22:01.585132 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.583787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46111 (local address 127.0.0.1:42768)
0619 10:22:01.585020 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.585073 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":39,"threads_started":1}
W20260619 10:22:01.587201 27189 catalog_manager.cc:4584] TS e26d0c143aea498ba7ca09e22c2bd066 (127.19.65.129:46111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88f43e21a0d74bf1a51ecd263c48a236: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46111: connect: Connection refused (error 111)
I20260619 10:22:01.592311 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27557
I20260619 10:22:01.595432 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:01.592180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:46685 (local address 127.0.0.1:43638)
0619 10:22:01.595374 (+ 3194us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:01.595406 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.130:46685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 10:22:01.597954 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27173
2026-06-19T10:22:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2852 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27811](info): set up 2 sockets
Jun 19 10:22:01 dist-test-slave-xd2q krb5kdc[27811](info): commencing operation
krb5kdc: starting...
W20260619 10:22:02.631994 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:22:02 dist-test-slave-xd2q krb5kdc[27811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:22:02.745900 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35367
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:02.822781 27824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:02.822965 27824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:02.822988 27824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:02.824162 27824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:02.824201 27824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:02.824214 27824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:02.824226 27824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:02.824239 27824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:02.825695 27824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:35367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:02.825934 27824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:02.826176 27824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:02.830737 27829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:02.830727 27832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:02.830894 27824 server_base.cc:1034] running on GCE node
W20260619 10:22:02.831044 27830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:02.831351 27824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:02.831709 27824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:02.832921 27824 hybrid_clock.cc:648] HybridClock initialized: now 1781864522832899 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T10:22:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1244 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:22:02 dist-test-slave-xd2q krb5kdc[27837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:22:02 dist-test-slave-xd2q krb5kdc[27837](info): set up 2 sockets
Jun 19 10:22:02 dist-test-slave-xd2q krb5kdc[27837](info): commencing operation
krb5kdc: starting...
W20260619 10:22:03.871737 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:22:03 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:22:03.981873 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39053
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:04.059079 27850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.059296 27850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.059350 27850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.060534 27850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:04.060575 27850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.060590 27850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.060602 27850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:04.060654 27850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:04.062175 27850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.062443 27850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.062662 27850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.067212 27858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.067242 27855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.067265 27856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.067412 27850 server_base.cc:1034] running on GCE node
I20260619 10:22:04.067621 27850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.067912 27850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.069056 27850 hybrid_clock.cc:648] HybridClock initialized: now 1781864524069057 us; error 41 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.071420 27850 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:22:04.071897 27850 webserver.cc:460] Webserver started at http://127.19.65.190:36655/ using document root <none> and password file <none>
I20260619 10:22:04.072131 27850 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.072187 27850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.072307 27850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:04.073266 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "081bbd8d54f649ba907a52605b2d9cbd"
server_key_iv: "44ec4a41e8588e75ccc9e13d486ebd7f"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.073606 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "081bbd8d54f649ba907a52605b2d9cbd"
server_key_iv: "44ec4a41e8588e75ccc9e13d486ebd7f"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.074887 27850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:22:04.075635 27865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.075806 27850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.075875 27850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "081bbd8d54f649ba907a52605b2d9cbd"
server_key_iv: "44ec4a41e8588e75ccc9e13d486ebd7f"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.075963 27850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.084954 27850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.086709 27850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.086881 27850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.090864 27850 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39053
I20260619 10:22:04.090924 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:39053 every 8 connection(s)
I20260619 10:22:04.091202 27850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:04.092160 27918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:04.094230 27918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Bootstrap starting.
I20260619 10:22:04.094959 27918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:04.095353 27918 log.cc:824] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:04.095597 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27850
I20260619 10:22:04.095705 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:22:04.095954 19718 external_mini_cluster.cc:1428] Setting key 223197a77edc6390ba50784a7107b697
I20260619 10:22:04.096119 27918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: No bootstrap required, opened a new log
I20260619 10:22:04.098212 27918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.098371 27918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:04.098412 27918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c98d63f7dde4dcdb5c4f5011c88ecc4, State: Initialized, Role: FOLLOWER
I20260619 10:22:04.098523 27918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.098606 27918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:04.098649 27918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:04.098693 27918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:04.099479 27918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.099579 27918 leader_election.cc:302] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c98d63f7dde4dcdb5c4f5011c88ecc4; no voters:
I20260619 10:22:04.099731 27918 leader_election.cc:288] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:22:04.099752 27923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:04.099874 27923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 LEADER]: Becoming Leader. State: Replica: 5c98d63f7dde4dcdb5c4f5011c88ecc4, State: Running, Role: LEADER
I20260619 10:22:04.099948 27923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.099984 27918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:04.100382 27925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c98d63f7dde4dcdb5c4f5011c88ecc4. Latest consensus state: current_term: 1 leader_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } } }
I20260619 10:22:04.100414 27924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } } }
I20260619 10:22:04.100559 27924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:04.100538 27925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:04.100804 27932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:04.101364 27932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:04.105876 27932 catalog_manager.cc:1347] Generated new cluster ID: ce21b7f6f2d545d7af9a02709237675d
I20260619 10:22:04.105928 27932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:22:04.108855 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.096687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.19.65.190:39053)
0619 10:22:04.097205 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.097214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.097245 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:22:04.098195 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.098201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.098246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.098394 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.099097 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.100244 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.101413 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.102126 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.102504 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.104744 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:04.104748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:04.104749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:04.104759 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:04.106670 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.107115 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.107118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.107118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.107245 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.107496 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.107496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.107497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.107801 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:04.108441 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:04.108510 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 10:22:04.108648 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":71,"threads_started":1}
I20260619 10:22:04.121587 27932 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:22:04.122056 27932 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:04.126886 27932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Generated new TSK 0
I20260619 10:22:04.127044 27932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:22:04.152050 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:04.230320 27944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.230508 27944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.230528 27944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.231659 27944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.231693 27944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.231717 27944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:04.233064 27944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.233264 27944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.233441 27944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.238126 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.238188 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.238191 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.238332 27944 server_base.cc:1034] running on GCE node
I20260619 10:22:04.238569 27944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.238824 27944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.239977 27944 hybrid_clock.cc:648] HybridClock initialized: now 1781864524239960 us; error 36 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.242396 27944 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:22:04.242805 27944 webserver.cc:460] Webserver started at http://127.19.65.129:37583/ using document root <none> and password file <none>
I20260619 10:22:04.243045 27944 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.243112 27944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.243243 27944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:04.244280 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "1b9c7264796d4e9d840bc15d5b140348"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "1cf8169c1092b75abb182043394f57eb"
server_key_iv: "65a0a0f56f5c5dca153b431d6909ca4d"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.244637 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "1b9c7264796d4e9d840bc15d5b140348"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "1cf8169c1092b75abb182043394f57eb"
server_key_iv: "65a0a0f56f5c5dca153b431d6909ca4d"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.245929 27944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:04.246651 27959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.246836 27944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:04.246932 27944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "1b9c7264796d4e9d840bc15d5b140348"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "1cf8169c1092b75abb182043394f57eb"
server_key_iv: "65a0a0f56f5c5dca153b431d6909ca4d"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.247017 27944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.255640 27944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.257328 27944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.257480 27944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.257684 27944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:04.257964 27944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:04.258018 27944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.258051 27944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:04.258077 27944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.263652 27944 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:36639
I20260619 10:22:04.263743 28072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:36639 every 8 connection(s)
I20260619 10:22:04.264009 27944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:22:04.266040 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 27944
I20260619 10:22:04.266132 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:22:04.266346 19718 external_mini_cluster.cc:1428] Setting key 36d23cb63ab89d7091320a6913657dc1
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:04.273610 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.264598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:37877)
0619 10:22:04.264772 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.264803 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.265283 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.265524 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.265527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.266072 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.266332 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.266337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.267095 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.267096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.267915 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.267917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.268386 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.268387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.268533 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.269007 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.269017 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.270570 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:04.272410 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.272415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.272416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.272617 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.272826 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.272827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.272827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.272914 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.273187 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:04.273190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:04.273290 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 10:22:04.273365 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:22:04.273468 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260619 10:22:04.273615 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.264623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:37877 (local address 127.19.65.190:39053)
0619 10:22:04.264682 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.264684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.264881 (+ 197us) server_negotiation.cc:407] Connection header received
0619 10:22:04.265394 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.265395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.265411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.265462 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.266444 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.267001 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.267998 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.268303 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.268362 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.270668 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:04.270670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:04.270670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:04.270679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:04.272337 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.272694 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.272695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.272695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.272758 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.272981 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.272982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.272983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.273119 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:04.273195 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:04.273412 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 10:22:04.273493 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:22:04.274158 28073 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39053
I20260619 10:22:04.274253 28073 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:04.274444 28073 heartbeater.cc:511] Master 127.19.65.190:39053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:22:04.274966 27882 ts_manager.cc:194] Registered new tserver with Master: 1b9c7264796d4e9d840bc15d5b140348 (127.19.65.129:36639)
I20260619 10:22:04.275636 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:37877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:22:04.292047 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:04.370668 28078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.370898 28078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.370934 28078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.372179 28078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.372241 28078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.372299 28078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:04.373737 28078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.373979 28078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.374197 28078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.379263 28083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.379215 28084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.379221 28086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.379586 28078 server_base.cc:1034] running on GCE node
I20260619 10:22:04.379781 28078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.380028 28078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.381206 28078 hybrid_clock.cc:648] HybridClock initialized: now 1781864524381192 us; error 24 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.383843 28078 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:22:04.384284 28078 webserver.cc:460] Webserver started at http://127.19.65.130:41135/ using document root <none> and password file <none>
I20260619 10:22:04.384452 28078 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.384491 28078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.384574 28078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:04.385517 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "c089b5f442da4a01988d0e7ff49cf44c"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "c35b66a826c69389daca7873aa653da2"
server_key_iv: "e093eec8f7dc8c12df8be28884701052"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.385800 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "c089b5f442da4a01988d0e7ff49cf44c"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "c35b66a826c69389daca7873aa653da2"
server_key_iv: "e093eec8f7dc8c12df8be28884701052"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.387053 28078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:22:04.387814 28093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.388000 28078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.388098 28078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "c089b5f442da4a01988d0e7ff49cf44c"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "c35b66a826c69389daca7873aa653da2"
server_key_iv: "e093eec8f7dc8c12df8be28884701052"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.388176 28078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.404394 28078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.406162 28078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.406347 28078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.406574 28078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:04.406864 28078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:04.406919 28078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.406958 28078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:04.406985 28078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.412799 28078 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:46197
I20260619 10:22:04.412834 28206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:46197 every 8 connection(s)
I20260619 10:22:04.413164 28078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:22:04.416481 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28078
I20260619 10:22:04.416566 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:22:04.416770 19718 external_mini_cluster.cc:1428] Setting key e9714c820cecb9a3f0e05259804f1788
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:04.422986 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.413835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:34463)
0619 10:22:04.413997 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.414034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.414503 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.414698 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.414701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.415242 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.415502 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.415508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.416170 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.416171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.417274 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.417277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.417768 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.417769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.417870 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.418352 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.418360 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.419848 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:04.421682 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.421688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.421690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.421903 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.422131 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.422132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.422132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.422229 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.422518 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:04.422524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:04.422643 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 10:22:04.422737 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 10:22:04.422844 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260619 10:22:04.423005 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.413846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34463 (local address 127.19.65.190:39053)
0619 10:22:04.413910 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.413912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.414115 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:22:04.414585 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.414586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.414594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.414640 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.415575 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.416057 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.417384 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.417684 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.417756 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.419918 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:04.419920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:04.419920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:04.419928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:04.421559 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.421984 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.421985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.421986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.422053 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.422296 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.422297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.422297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.422445 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:04.422522 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:04.422788 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 10:22:04.422874 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:22:04.423555 28207 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39053
I20260619 10:22:04.423655 28207 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:04.423816 28207 heartbeater.cc:511] Master 127.19.65.190:39053 requested a full tablet report, sending...
I20260619 10:22:04.424194 27882 ts_manager.cc:194] Registered new tserver with Master: c089b5f442da4a01988d0e7ff49cf44c (127.19.65.130:46197)
I20260619 10:22:04.424690 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:34463
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:22:04.441804 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:04.520349 28212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.520543 28212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.520565 28212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.521694 28212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.521735 28212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.521760 28212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:04.523078 28212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.523296 28212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.523540 28212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.528030 28217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.528004 28218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.528110 28220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.528375 28212 server_base.cc:1034] running on GCE node
I20260619 10:22:04.528560 28212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.528775 28212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.529923 28212 hybrid_clock.cc:648] HybridClock initialized: now 1781864524529918 us; error 35 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.532200 28212 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:22:04.532636 28212 webserver.cc:460] Webserver started at http://127.19.65.131:34883/ using document root <none> and password file <none>
I20260619 10:22:04.532845 28212 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.532908 28212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.533052 28212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:04.534039 28212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "ce7c478e93dd4bbcbe35e07d3048e264"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "b2ca5af1a434d1deac71cc4d6f26c32c"
server_key_iv: "6638cad9aa0aa2f59d760b453d79b753"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.534368 28212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "ce7c478e93dd4bbcbe35e07d3048e264"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "b2ca5af1a434d1deac71cc4d6f26c32c"
server_key_iv: "6638cad9aa0aa2f59d760b453d79b753"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.535717 28212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:22:04.536463 28227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.536633 28212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:04.536716 28212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "ce7c478e93dd4bbcbe35e07d3048e264"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "b2ca5af1a434d1deac71cc4d6f26c32c"
server_key_iv: "6638cad9aa0aa2f59d760b453d79b753"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.536793 28212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.544066 28212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.545719 28212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.545858 28212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.546046 28212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:04.546325 28212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:04.546375 28212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.546411 28212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:04.546437 28212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.552309 28212 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40579
I20260619 10:22:04.552348 28340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:40579 every 8 connection(s)
I20260619 10:22:04.552639 28212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:22:04.555859 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28212
I20260619 10:22:04.555986 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:22:04.556206 19718 external_mini_cluster.cc:1428] Setting key 98e070db8e1efbf4865be667450ce906
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:04.561861 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.553276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40883 (local address 127.19.65.190:39053)
0619 10:22:04.553347 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.553348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.553502 (+ 154us) server_negotiation.cc:407] Connection header received
0619 10:22:04.553959 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.553960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.553972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.554049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.555045 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.555620 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.557057 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.557259 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.557309 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.559494 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:04.559496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:04.559496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:04.559502 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:04.560701 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.561105 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.561106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.561106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.561159 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:04.561376 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:04.561376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:04.561376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:04.561464 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:04.561510 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:04.561744 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 10:22:04.561800 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:22:04.562132 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.553231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:40883)
0619 10:22:04.553384 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.553421 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.553866 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.554185 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.554189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.554740 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.554954 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.554959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.555706 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.555707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.556987 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.556990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.557327 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.557327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.557423 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.557949 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.557957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.559397 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:04.560802 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.560807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.560809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.561023 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.561219 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:04.561219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:04.561219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:04.561319 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:04.561517 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:04.561520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:04.561630 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 10:22:04.561720 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 10:22:04.561830 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260619 10:22:04.562598 28341 heartbeater.cc:347] Connected to a master server at 127.19.65.190:39053
I20260619 10:22:04.562677 28341 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:04.562863 28341 heartbeater.cc:511] Master 127.19.65.190:39053 requested a full tablet report, sending...
I20260619 10:22:04.563416 27882 ts_manager.cc:194] Registered new tserver with Master: ce7c478e93dd4bbcbe35e07d3048e264 (127.19.65.131:40579)
I20260619 10:22:04.563743 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:40883
I20260619 10:22:04.567863 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
Entry for principal oryx/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab.
WARNING: no policy specified for oryx/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
Entry for principal oryx/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab.
WARNING: no policy specified for oryx/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
Entry for principal oryx/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab.
I20260619 10:22:04.634424 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27944
I20260619 10:22:04.638747 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28078
I20260619 10:22:04.643095 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28212
I20260619 10:22:04.647373 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 27850
I20260619 10:22:04.652197 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39053
--webserver_interface=127.19.65.190
--webserver_port=36655
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:04.729378 28350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.729558 28350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.729579 28350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.730688 28350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:04.730727 28350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.730739 28350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.730751 28350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:04.730798 28350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:04.732152 28350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=36655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.732383 28350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.732591 28350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.737285 28355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.737294 28358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.737320 28356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.737507 28350 server_base.cc:1034] running on GCE node
I20260619 10:22:04.737756 28350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.738075 28350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.739228 28350 hybrid_clock.cc:648] HybridClock initialized: now 1781864524739203 us; error 42 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.741671 28350 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:22:04.742069 28350 webserver.cc:460] Webserver started at http://127.19.65.190:36655/ using document root <none> and password file <none>
I20260619 10:22:04.742345 28350 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.742416 28350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.743821 28350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:04.744513 28365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.744700 28350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:04.744798 28350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "081bbd8d54f649ba907a52605b2d9cbd"
server_key_iv: "44ec4a41e8588e75ccc9e13d486ebd7f"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.745143 28350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.769578 28350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.771852 28350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.772109 28350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.776026 28350 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39053
I20260619 10:22:04.776068 28417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:39053 every 8 connection(s)
I20260619 10:22:04.776444 28350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:04.776587 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28350
I20260619 10:22:04.776887 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:36639
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:39053
--webserver_port=37583
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:22:04.777933 28418 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:22:04.778589 28418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Bootstrap starting.
I20260619 10:22:04.786194 28418 log.cc:824] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:04.787848 28418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:22:04.788118 28418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Bootstrap complete.
I20260619 10:22:04.790230 28418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.790465 28418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c98d63f7dde4dcdb5c4f5011c88ecc4, State: Initialized, Role: FOLLOWER
I20260619 10:22:04.790642 28418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.790716 28418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:04.790767 28418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:04.790818 28418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:22:04.791831 28418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.791949 28418 leader_election.cc:302] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c98d63f7dde4dcdb5c4f5011c88ecc4; no voters:
I20260619 10:22:04.792122 28418 leader_election.cc:288] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:22:04.792181 28422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:22:04.792330 28422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [term 2 LEADER]: Becoming Leader. State: Replica: 5c98d63f7dde4dcdb5c4f5011c88ecc4, State: Running, Role: LEADER
I20260619 10:22:04.792369 28418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:04.792423 28422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } }
I20260619 10:22:04.792655 28422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c98d63f7dde4dcdb5c4f5011c88ecc4. Latest consensus state: current_term: 2 leader_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } } }
I20260619 10:22:04.792713 28422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:04.792671 28423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c98d63f7dde4dcdb5c4f5011c88ecc4" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39053 } } }
I20260619 10:22:04.792884 28427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:04.792912 28423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:04.793316 28427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:22:04.793435 28427 catalog_manager.cc:1259] Loaded cluster ID: ce21b7f6f2d545d7af9a02709237675d
I20260619 10:22:04.793469 28427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:22:04.794632 28427 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:04.794801 28427 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c98d63f7dde4dcdb5c4f5011c88ecc4: Loaded TSK: 0
I20260619 10:22:04.795017 28427 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:22:04.859609 28420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:04.859788 28420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:04.859809 28420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:04.860968 28420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:04.861014 28420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:04.861037 28420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:04.862387 28420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.129.keytab
--principal=oryx/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:36639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=37583
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:04.862592 28420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:04.862826 28420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:04.868518 28445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.868551 28447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:04.868777 28444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:04.868878 28420 server_base.cc:1034] running on GCE node
I20260619 10:22:04.869092 28420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:04.869338 28420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:04.870494 28420 hybrid_clock.cc:648] HybridClock initialized: now 1781864524870481 us; error 36 us; skew 500 ppm
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864524, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:04.873078 28420 init.cc:377] Logged in from keytab as oryx/127.19.65.129@KRBTEST.COM (short username oryx)
I20260619 10:22:04.873572 28420 webserver.cc:460] Webserver started at http://127.19.65.129:37583/ using document root <none> and password file <none>
I20260619 10:22:04.873781 28420 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:04.873869 28420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:04.875145 28420 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:04.875767 28454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.875985 28420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:22:04.876092 28420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "1b9c7264796d4e9d840bc15d5b140348"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "1cf8169c1092b75abb182043394f57eb"
server_key_iv: "65a0a0f56f5c5dca153b431d6909ca4d"
server_key_version: "encryptionkey@0"
I20260619 10:22:04.876474 28420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:04.902456 28420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:04.905031 28420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:04.905176 28420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:04.905440 28420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:04.905766 28420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:04.905821 28420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.905861 28420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:04.905890 28420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:04.911978 28420 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:36639
I20260619 10:22:04.912042 28567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:36639 every 8 connection(s)
I20260619 10:22:04.912349 28420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:04.921635 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28420
I20260619 10:22:04.921932 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:46197
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:39053
--webserver_port=41135
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:04.923261 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.913133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:44805)
0619 10:22:04.913319 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.913364 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.913813 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.915865 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.915868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.916410 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.916606 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.916611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.918315 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.918316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.919084 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.919086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.919962 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.919963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.920057 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.920606 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.920614 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.923069 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
W20260619 10:22:04.923535 28568 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:39053: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
W20260619 10:22:04.923585 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:04.913189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44805 (local address 127.19.65.190:39053)
0619 10:22:04.915429 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.915438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.915477 (+ 39us) server_negotiation.cc:407] Connection header received
0619 10:22:04.915599 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.915605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.915667 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.915807 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.916731 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.918213 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.919172 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.919880 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.919988 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.923417 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44805: BlockingRecv error: recv got EOF from 127.19.65.129:44805 (error 108)
Metrics: {"server-negotiator.queue_time_us":2185,"thread_start_us":101,"threads_started":1}
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:04.928954 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.923724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:59273)
0619 10:22:04.923762 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.923772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.923803 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.923969 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.923969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.924109 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.924164 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.924177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.924898 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.924899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.925729 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.925731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.926257 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.926258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.926323 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.926815 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.926820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.928895 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 10:22:04.929267 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:04.923806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:59273 (local address 127.19.65.190:39053)
0619 10:22:04.923840 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.923843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.923851 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:22:04.923874 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.923875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.923888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.923953 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.924261 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.924828 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.925818 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.926166 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.926238 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.929192 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:59273: BlockingRecv error: recv got EOF from 127.19.65.129:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:04 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:04.933874 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:04.929163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:43315)
0619 10:22:04.929427 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:04.929437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:04.929473 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:04.929643 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:04.929644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:04.929787 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:04.929817 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.929819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.930744 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.930745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.931230 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:04.931231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.931767 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.931768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:04.931805 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.932243 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:04.932247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:04.933805 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":81,"threads_started":1}
W20260619 10:22:04.934007 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:04.929323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:43315 (local address 127.19.65.190:39053)
0619 10:22:04.929507 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:04.929509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:04.929519 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:04.929540 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:04.929541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:04.929554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:04.929622 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:04.929898 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.930657 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.931330 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:04.931683 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:04.931773 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:04.933879 (+ 2106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:43315: BlockingRecv error: recv got EOF from 127.19.65.129:43315 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
W20260619 10:22:04.934269 28568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:22:05.003372 28572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:05.003553 28572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:05.003576 28572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:05.004746 28572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:05.004787 28572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:05.004838 28572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:05.006235 28572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.130.keytab
--principal=oryx/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=41135
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:05.006459 28572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:05.006692 28572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:05.011341 28579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:05.011360 28580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:05.011360 28582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:05.011548 28572 server_base.cc:1034] running on GCE node
I20260619 10:22:05.011761 28572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:05.011998 28572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:05.013152 28572 hybrid_clock.cc:648] HybridClock initialized: now 1781864525013136 us; error 28 us; skew 500 ppm
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864525, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:05.015527 28572 init.cc:377] Logged in from keytab as oryx/127.19.65.130@KRBTEST.COM (short username oryx)
I20260619 10:22:05.015956 28572 webserver.cc:460] Webserver started at http://127.19.65.130:41135/ using document root <none> and password file <none>
I20260619 10:22:05.016168 28572 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:05.016253 28572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:05.017554 28572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:05.018079 28589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.018239 28572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:05.018335 28572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "c089b5f442da4a01988d0e7ff49cf44c"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "c35b66a826c69389daca7873aa653da2"
server_key_iv: "e093eec8f7dc8c12df8be28884701052"
server_key_version: "encryptionkey@0"
I20260619 10:22:05.018646 28572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:05.039129 28572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:05.040930 28572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:05.041081 28572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:05.041286 28572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:05.041569 28572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:05.041620 28572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.041661 28572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:05.041687 28572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.047055 28572 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:46197
I20260619 10:22:05.047089 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:46197 every 8 connection(s)
I20260619 10:22:05.047566 28572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.054497 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.048230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:52903)
0619 10:22:05.048383 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.048432 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.048877 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.049152 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.049156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.049687 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.049849 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.049854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.050509 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.050510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.051325 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.051327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.051728 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.051729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.051842 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.052300 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.052307 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.054373 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260619 10:22:05.054504 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.048265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52903 (local address 127.19.65.190:39053)
0619 10:22:05.048306 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.048308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.048531 (+ 223us) server_negotiation.cc:407] Connection header received
0619 10:22:05.048997 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.048998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.049016 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.049065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.049947 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.050420 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.051430 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.051626 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.051722 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.054460 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:52903: BlockingRecv error: recv got EOF from 127.19.65.130:52903 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:22:05.054744 28703 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:39053: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
I20260619 10:22:05.057791 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28572
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.058087 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:40579
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:39053
--webserver_port=34883
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.058899 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.054960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:46635)
0619 10:22:05.055026 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.055038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.055086 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.055248 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.055249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.055410 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.055449 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.055450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.056006 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.056006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.056494 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.056495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.056841 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.056842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.056885 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.057243 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.057247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.058804 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 10:22:05.058914 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.055037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:46635 (local address 127.19.65.190:39053)
0619 10:22:05.055099 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.055101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.055109 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:22:05.055131 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.055132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.055174 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.055228 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.055511 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.055944 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.056579 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.056754 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.056832 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.058878 (+ 2046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:46635: BlockingRecv error: recv got EOF from 127.19.65.130:46635 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.063591 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.059118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:46841)
0619 10:22:05.059262 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.059271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.059344 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.059480 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.059480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.059626 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.059659 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.059660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.060535 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.060536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.060973 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.060974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.061425 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.061426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.061458 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.061846 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.061852 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.063551 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260619 10:22:05.063735 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.059171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:46841 (local address 127.19.65.190:39053)
0619 10:22:05.059197 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.059199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.059337 (+ 138us) server_negotiation.cc:407] Connection header received
0619 10:22:05.059400 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.059401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.059411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.059476 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.059731 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.060456 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.061051 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.061353 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.061424 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.063653 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:46841: BlockingRecv error: recv got EOF from 127.19.65.130:46841 (error 108)
Metrics: {"server-negotiator.queue_time_us":11}
W20260619 10:22:05.063792 28703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:22:05.138420 28707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:05.138626 28707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:05.138672 28707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:05.139812 28707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:05.139902 28707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:05.139948 28707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:05.141304 28707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/krb5kdc/oryx_127.19.65.131.keytab
--principal=oryx/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=34883
--tserver_master_addrs=127.19.65.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:05.141566 28707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:05.141799 28707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:05.146440 28713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:05.146497 28712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:05.146615 28707 server_base.cc:1034] running on GCE node
W20260619 10:22:05.146493 28715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:05.146852 28707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:05.147056 28707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:05.148231 28707 hybrid_clock.cc:648] HybridClock initialized: now 1781864525148218 us; error 30 us; skew 500 ppm
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864525, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:05.150571 28707 init.cc:377] Logged in from keytab as oryx/127.19.65.131@KRBTEST.COM (short username oryx)
I20260619 10:22:05.151041 28707 webserver.cc:460] Webserver started at http://127.19.65.131:34883/ using document root <none> and password file <none>
I20260619 10:22:05.151368 28707 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:05.151427 28707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:05.152695 28707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:05.153337 28722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.153515 28707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:05.153609 28707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "ce7c478e93dd4bbcbe35e07d3048e264"
format_stamp: "Formatted at 2026-06-19 10:22:04 on dist-test-slave-xd2q"
server_key: "b2ca5af1a434d1deac71cc4d6f26c32c"
server_key_iv: "6638cad9aa0aa2f59d760b453d79b753"
server_key_version: "encryptionkey@0"
I20260619 10:22:05.153935 28707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:05.188752 28707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:05.190733 28707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:05.190900 28707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:05.191169 28707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:05.191489 28707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:05.191546 28707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.191598 28707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:05.191648 28707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:05.197737 28707 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:40579
I20260619 10:22:05.197784 28835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:40579 every 8 connection(s)
I20260619 10:22:05.198141 28707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:22:05.203711 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28707
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:22:05.205890 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.198899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:45801 (local address 127.19.65.190:39053)
0619 10:22:05.198948 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.198950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.199160 (+ 210us) server_negotiation.cc:407] Connection header received
0619 10:22:05.199715 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.199717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.199731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.199784 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.200741 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.201235 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.202354 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.202568 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.202663 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.205832 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:45801: BlockingRecv error: recv got EOF from 127.19.65.131:45801 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:22:05.205964 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.198891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:45801)
0619 10:22:05.199030 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.199065 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.199604 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.199862 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.199867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.200431 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.200625 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.200631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.201336 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.201337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.202260 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.202262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.202671 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.202672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.202795 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.203282 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.203431 (+ 149us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.205765 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260619 10:22:05.206203 28836 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.65.190:39053: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.210932 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.206417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:58559)
0619 10:22:05.206537 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.206549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.206599 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.206736 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.206736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.206862 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.206905 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.206906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.207579 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.207580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.208185 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.208188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.208534 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.208535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.208600 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.209140 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.209144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.210841 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 10:22:05.210995 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.206491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:58559 (local address 127.19.65.190:39053)
0619 10:22:05.206540 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.206542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.206585 (+ 43us) server_negotiation.cc:407] Connection header received
0619 10:22:05.206678 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.206678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.206687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.206734 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.206978 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.207493 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.208269 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.208452 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.208533 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.210932 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:58559: BlockingRecv error: recv got EOF from 127.19.65.131:58559 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:22:05.212088 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.204314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.19.65.190:39053)
0619 10:22:05.204361 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.204362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.204389 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:22:05.204470 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.204472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.204480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.204572 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.205212 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.205697 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.206851 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.207055 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.207131 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.208408 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:05.208412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:05.208412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:05.208424 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:05.210274 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:05.210670 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:05.210674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:05.210674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:05.210789 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:05.211333 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:05.211334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:05.211335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:05.211635 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:05.211780 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:05.211901 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 10:22:05.211985 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.215366 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.211190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:50607)
0619 10:22:05.211286 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.211296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.211352 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.211457 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.211458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.211581 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.211610 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.211611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.212455 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.212456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.213086 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.213088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.213363 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.213363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.213407 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.213796 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.213800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.215278 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 10:22:05.215409 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.211232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:50607 (local address 127.19.65.190:39053)
0619 10:22:05.211311 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.211313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.211348 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:22:05.211406 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.211406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.211415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.211453 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.211707 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.212389 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.213146 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.213310 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.213354 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.215354 (+ 2000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:50607: BlockingRecv error: recv got EOF from 127.19.65.131:50607 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 10:22:05.215643 28836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:05 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:05.939252 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:05.934652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:32951)
0619 10:22:05.934795 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:05.934806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:05.934862 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:05.935016 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:05.935017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:05.935166 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:05.935213 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.935215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.935896 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.935897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.936426 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:05.936428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.936789 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.936790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:05.936857 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.937391 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:05.937396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:05.939188 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
W20260619 10:22:05.939384 28841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:05.934737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:32951 (local address 127.19.65.190:39053)
0619 10:22:05.934877 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:05.934879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:05.934888 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:05.934913 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:05.934914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:05.934933 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:05.934995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:05.935325 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.935829 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.936501 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:05.936697 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:05.936769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:05.939268 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:32951: BlockingRecv error: recv got EOF from 127.19.65.129:32951 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:06.069027 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:06.064210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:58763)
0619 10:22:06.064378 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:06.064397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:06.064484 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:06.064667 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:06.064668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:06.064847 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:06.064893 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.064894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.065645 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.065646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.066201 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.066203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.066571 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.066572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.066643 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.067127 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:06.067137 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:06.068926 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
W20260619 10:22:06.069070 28841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:06.064293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:58763 (local address 127.19.65.190:39053)
0619 10:22:06.064376 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:06.064378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:06.064469 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:22:06.064560 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:06.064561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:06.064574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:06.064648 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:06.064978 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.065556 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.066292 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.066495 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.066573 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.069006 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:58763: BlockingRecv error: recv got EOF from 127.19.65.130:58763 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:06.220811 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:06.216004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:55945)
0619 10:22:06.216157 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:06.216169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:06.216220 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:06.216340 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:06.216340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:06.216486 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:06.216531 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.216533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.217228 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.217229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.218028 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.218031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.218359 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.218359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.218398 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.218876 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:06.218881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:06.220703 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
W20260619 10:22:06.220839 28841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:06.216089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:55945 (local address 127.19.65.190:39053)
0619 10:22:06.216197 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:06.216199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:06.216225 (+ 26us) server_negotiation.cc:407] Connection header received
0619 10:22:06.216268 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:06.216268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:06.216278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:06.216321 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:06.216593 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.217142 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.218100 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.218291 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.218347 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.220778 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:55945: BlockingRecv error: recv got EOF from 127.19.65.131:55945 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:06 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:06.944186 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:06.939733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:55727)
0619 10:22:06.939889 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:06.939900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:06.939950 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:06.940120 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:06.940120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:06.940270 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:06.940318 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.940320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.941003 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.941004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.941514 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:06.941515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.941849 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.941850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:06.941887 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.942336 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:06.942341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:06.944133 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260619 10:22:06.944324 28845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:06.939835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:55727 (local address 127.19.65.190:39053)
0619 10:22:06.939969 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:06.939972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:06.939981 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:06.940032 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:06.940032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:06.940046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:06.940107 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:06.940384 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.940938 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.941580 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:06.941787 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:06.941840 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:06.944226 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:55727: BlockingRecv error: recv got EOF from 127.19.65.129:55727 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:07.073732 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:07.069493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:34095)
0619 10:22:07.069623 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:07.069633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:07.069685 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:07.069811 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:07.069812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:07.069951 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:07.069997 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.069999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.070578 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.070578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.071081 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.071082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.071418 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.071420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.071487 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.071912 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:07.071916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:07.073678 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
W20260619 10:22:07.073859 28845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:07.069571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34095 (local address 127.19.65.190:39053)
0619 10:22:07.069632 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:07.069634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:07.069676 (+ 42us) server_negotiation.cc:407] Connection header received
0619 10:22:07.069728 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:07.069728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:07.069747 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:07.069798 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:07.070059 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.070517 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.071150 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.071338 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.071417 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.073778 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:34095: BlockingRecv error: recv got EOF from 127.19.65.130:34095 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:07.226347 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:07.221291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:42271)
0619 10:22:07.221441 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:07.221478 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:07.221572 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:07.221748 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:07.221750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:07.221977 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:07.222038 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.222040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.222760 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.222761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.223412 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.223414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.223794 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.223795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.223853 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.224359 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:07.224365 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:07.226289 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":54,"threads_started":1}
W20260619 10:22:07.226491 28845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:07.221439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:42271 (local address 127.19.65.190:39053)
0619 10:22:07.221503 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:07.221505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:07.221549 (+ 44us) server_negotiation.cc:407] Connection header received
0619 10:22:07.221644 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:07.221644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:07.221660 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:07.221743 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:07.222168 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.222665 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.223509 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.223706 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.223794 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.226416 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:42271: BlockingRecv error: recv got EOF from 127.19.65.131:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:07 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:07.949209 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:07.944691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:44705)
0619 10:22:07.944821 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:07.944830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:07.944875 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:07.945002 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:07.945002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:07.945152 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:07.945198 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.945199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.945832 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.945833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.946341 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:07.946343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.946690 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.946691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:07.946734 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.947230 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:07.947235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:07.949155 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260619 10:22:07.949353 28849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:07.944743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44705 (local address 127.19.65.190:39053)
0619 10:22:07.944873 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:07.944875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:07.944890 (+ 15us) server_negotiation.cc:407] Connection header received
0619 10:22:07.944914 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:07.944914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:07.944925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:07.944991 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:07.945269 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.945760 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.946424 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:07.946619 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:07.946680 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:07.949266 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44705: BlockingRecv error: recv got EOF from 127.19.65.129:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:08.078830 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:08.074216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:59633)
0619 10:22:08.074354 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:08.074366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:08.074443 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:08.074587 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:08.074588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:08.074734 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:08.074778 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.074780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.075480 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.075481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.075994 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.075996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.076337 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.076338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.076412 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.076865 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:08.076869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:08.078738 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260619 10:22:08.078873 28849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:08.074284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59633 (local address 127.19.65.190:39053)
0619 10:22:08.074358 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:08.074360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:08.074446 (+ 86us) server_negotiation.cc:407] Connection header received
0619 10:22:08.074488 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:08.074489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:08.074507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:08.074577 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:08.074876 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.075392 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.076097 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.076270 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.076328 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.078811 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:59633: BlockingRecv error: recv got EOF from 127.19.65.130:59633 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:08.231528 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:08.226930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:48221)
0619 10:22:08.227098 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:08.227109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:08.227171 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:08.227315 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:08.227316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:08.227508 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:08.227554 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.227555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.228205 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.228206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.228741 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.228742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.229097 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.229098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.229160 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.229569 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:08.229573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:08.231435 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
W20260619 10:22:08.231565 28849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:08.227018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48221 (local address 127.19.65.190:39053)
0619 10:22:08.227089 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:08.227092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:08.227174 (+ 82us) server_negotiation.cc:407] Connection header received
0619 10:22:08.227224 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:08.227225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:08.227237 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:08.227308 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:08.227630 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.228115 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.228821 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.229012 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.229095 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.231505 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:48221: BlockingRecv error: recv got EOF from 127.19.65.131:48221 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:08 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:08.957089 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:08.949628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:33289)
0619 10:22:08.949757 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:08.949768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:08.949822 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:08.949971 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:08.949972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:08.950118 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:08.950165 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.950166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.950789 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.950790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.951333 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:08.951335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.951661 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.951662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:08.951703 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.955024 (+ 3321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:08.955029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:08.957003 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
W20260619 10:22:08.957285 28853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:08.949713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33289 (local address 127.19.65.190:39053)
0619 10:22:08.949837 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:08.949839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:08.949853 (+ 14us) server_negotiation.cc:407] Connection header received
0619 10:22:08.949886 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:08.949887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:08.949896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:08.949956 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:08.950233 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.950699 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.951409 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:08.951595 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:08.951646 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:08.957167 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:33289: BlockingRecv error: recv got EOF from 127.19.65.129:33289 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:09.083998 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:09.079357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:60535)
0619 10:22:09.079487 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:09.079504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:09.079572 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:09.079694 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:09.079694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:09.079852 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:09.079899 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.079901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.080613 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.080614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.081173 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.081175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.081529 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.081531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.081578 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.082109 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:09.082119 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:09.083940 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
W20260619 10:22:09.084139 28853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:09.079430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60535 (local address 127.19.65.190:39053)
0619 10:22:09.079506 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:09.079508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:09.079562 (+ 54us) server_negotiation.cc:407] Connection header received
0619 10:22:09.079617 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:09.079617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:09.079627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:09.079677 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:09.079957 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.080492 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.081262 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.081458 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.081510 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.084033 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:60535: BlockingRecv error: recv got EOF from 127.19.65.130:60535 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:09.236806 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:09.231977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:47917)
0619 10:22:09.232166 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:09.232181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:09.232257 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:09.232404 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:09.232405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:09.232582 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:09.232640 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.232642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.233289 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.233291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.233846 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.233847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.234236 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.234237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.234297 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.234802 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:09.234808 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:09.236749 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
W20260619 10:22:09.236933 28853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:09.232073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:47917 (local address 127.19.65.190:39053)
0619 10:22:09.232185 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:09.232187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:09.232239 (+ 52us) server_negotiation.cc:407] Connection header received
0619 10:22:09.232312 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:09.232313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:09.232327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:09.232399 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:09.232726 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.233204 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.233933 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.234129 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.234236 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.236851 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:47917: BlockingRecv error: recv got EOF from 127.19.65.131:47917 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:09 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:09.961875 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:09.957604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:38189)
0619 10:22:09.957755 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:09.957765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:09.957814 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:09.957935 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:09.957936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:09.958107 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:09.958151 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.958153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.958766 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.958767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.959418 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:09.959420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.959768 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.959768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:09.959807 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.960222 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:09.960227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:09.961821 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260619 10:22:09.961983 28857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:09.957717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:38189 (local address 127.19.65.190:39053)
0619 10:22:09.957839 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:09.957840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:09.957848 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:22:09.957866 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:09.957867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:09.957876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:09.957930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:09.958213 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.958701 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.959500 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:09.959700 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:09.959755 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:09.961907 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:38189: BlockingRecv error: recv got EOF from 127.19.65.129:38189 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:10.089716 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:10.084583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:46777)
0619 10:22:10.084759 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:10.084773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:10.084840 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:10.085037 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:10.085039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:10.085260 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:10.085309 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.085311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.085986 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.085988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.086633 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.086636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.087112 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.087113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.087175 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.087732 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:10.087737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:10.089659 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
W20260619 10:22:10.089893 28857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:10.084731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:46777 (local address 127.19.65.190:39053)
0619 10:22:10.084806 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:10.084808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:10.084821 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:22:10.084939 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:10.084940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:10.084957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:10.085033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:10.085398 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.085895 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.086742 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.086994 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.087110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.089811 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:46777: BlockingRecv error: recv got EOF from 127.19.65.130:46777 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:10.242215 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:10.237193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:52955)
0619 10:22:10.237374 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:10.237387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:10.237484 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:10.237679 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:10.237680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:10.237861 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:10.237907 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.237909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.238658 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.238659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.239216 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.239217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.239603 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.239604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.239669 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.240216 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:10.240221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:10.242114 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
W20260619 10:22:10.242274 28857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:10.237270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:52955 (local address 127.19.65.190:39053)
0619 10:22:10.237366 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:10.237368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:10.237465 (+ 97us) server_negotiation.cc:407] Connection header received
0619 10:22:10.237568 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:10.237569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:10.237583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:10.237663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:10.237997 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.238556 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.239323 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.239529 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.239603 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.242190 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:52955: BlockingRecv error: recv got EOF from 127.19.65.131:52955 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:10 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:10.966945 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:10.962327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:46259)
0619 10:22:10.962474 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:10.962484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:10.962537 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:10.962720 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:10.962721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:10.962863 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:10.962908 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.962910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.963575 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.963576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.964126 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:10.964128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.964452 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.964452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:10.964489 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.964930 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:10.964934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:10.966848 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
W20260619 10:22:10.967015 28861 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:10.962429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:46259 (local address 127.19.65.190:39053)
0619 10:22:10.962559 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:10.962560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:10.962567 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:22:10.962638 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:10.962638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:10.962648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:10.962700 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:10.962965 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.963501 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.964201 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:10.964382 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:10.964441 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:10.966943 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:46259: BlockingRecv error: recv got EOF from 127.19.65.129:46259 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:11.094880 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:11.090181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:50347)
0619 10:22:11.090323 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:11.090333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:11.090390 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:11.090530 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:11.090530 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:11.090683 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:11.090729 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.090731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.091401 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.091401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.091987 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.091989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.092283 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.092284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.092317 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.092707 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:11.092712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:11.094774 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
W20260619 10:22:11.094920 28861 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:11.090273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50347 (local address 127.19.65.190:39053)
0619 10:22:11.090351 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:11.090353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:11.090375 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:22:11.090451 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:11.090451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:11.090462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:11.090509 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:11.090787 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.091275 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.092045 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.092224 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.092274 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.094859 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:50347: BlockingRecv error: recv got EOF from 127.19.65.130:50347 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:11.247740 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:11.242740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:58363)
0619 10:22:11.242851 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:11.242862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:11.242921 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:11.243204 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:11.243205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:11.243389 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:11.243438 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.243440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.244086 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.244087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.244615 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.244617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.245126 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.245127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.245213 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.245631 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:11.245635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:11.247643 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260619 10:22:11.247777 28861 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:11.242814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:58363 (local address 127.19.65.190:39053)
0619 10:22:11.242977 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:11.242979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:11.242988 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:11.243017 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:11.243018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:11.243031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:11.243086 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:11.243503 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.243995 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.244756 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.245056 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.245118 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.247718 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:58363: BlockingRecv error: recv got EOF from 127.19.65.131:58363 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:11 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:11.972307 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:11.967579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:45369)
0619 10:22:11.967870 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:11.967881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:11.967951 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:11.968108 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:11.968109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:11.968260 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:11.968305 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.968306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.968932 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.968933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.969542 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:11.969545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.969903 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.969904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:11.969942 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.970462 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:11.970467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:11.972251 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":245,"threads_started":1}
W20260619 10:22:11.972448 28864 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:11.967686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45369 (local address 127.19.65.190:39053)
0619 10:22:11.967850 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:11.967852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:11.967972 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:22:11.967999 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:11.968000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:11.968011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:11.968098 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:11.968369 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.968862 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.969617 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:11.969829 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:11.969905 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:11.972356 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:45369: BlockingRecv error: recv got EOF from 127.19.65.129:45369 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:12.099850 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:12.095482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:42219)
0619 10:22:12.095623 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:12.095636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:12.095706 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:12.095831 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:12.095832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:12.096017 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:12.096072 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.096074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.096693 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.096694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.097168 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.097170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.097508 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.097509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.097562 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.097980 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:12.097985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:12.099749 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
W20260619 10:22:12.099887 28864 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:12.095571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:42219 (local address 127.19.65.190:39053)
0619 10:22:12.095641 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:12.095643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:12.095689 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:22:12.095761 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:12.095762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:12.095776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:12.095827 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:12.096159 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.096610 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.097249 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.097442 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.097510 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.099825 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:42219: BlockingRecv error: recv got EOF from 127.19.65.130:42219 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:12.252682 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:12.248189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:52321)
0619 10:22:12.248342 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:12.248355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:12.248426 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:12.248588 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:12.248588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:12.248744 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:12.248791 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.248792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.249422 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.249422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.249936 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.249937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.250255 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.250255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.250294 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.250762 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:12.250770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:12.252625 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
W20260619 10:22:12.252791 28864 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:12.248295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:52321 (local address 127.19.65.190:39053)
0619 10:22:12.248410 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:12.248412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:12.248426 (+ 14us) server_negotiation.cc:407] Connection header received
0619 10:22:12.248484 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:12.248485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:12.248499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:12.248566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:12.248862 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.249343 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.250008 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.250190 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.250246 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.252722 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:52321: BlockingRecv error: recv got EOF from 127.19.65.131:52321 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:12 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:12.977135 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:12.972747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:58007)
0619 10:22:12.972890 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:12.972900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:12.972946 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:12.973148 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:12.973149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:12.973298 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:12.973346 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.973348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.973964 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.973965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.974460 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:12.974461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.974785 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.974786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:12.974842 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.975255 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:12.975259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:12.977033 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260619 10:22:12.977192 28869 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:12.972854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:58007 (local address 127.19.65.190:39053)
0619 10:22:12.972980 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:12.972982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:12.972988 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:22:12.973008 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:12.973008 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:12.973017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:12.973125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:12.973420 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.973893 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.974530 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:12.974712 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:12.974772 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:12.977120 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:58007: BlockingRecv error: recv got EOF from 127.19.65.129:58007 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:13.104779 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:13.100348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:52451)
0619 10:22:13.100464 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:13.100474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:13.100537 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:13.100667 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:13.100668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:13.100823 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:13.100869 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.100871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.101654 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.101655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.102158 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.102160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.102477 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.102478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.102516 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.102937 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:13.102941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:13.104721 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260619 10:22:13.104915 28869 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:13.100368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52451 (local address 127.19.65.190:39053)
0619 10:22:13.100432 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:13.100435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:13.100518 (+ 83us) server_negotiation.cc:407] Connection header received
0619 10:22:13.100591 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:13.100592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:13.100602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:13.100646 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:13.100933 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.101577 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.102236 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.102414 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.102466 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.104833 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:52451: BlockingRecv error: recv got EOF from 127.19.65.130:52451 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:13.257988 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:13.253183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:60635)
0619 10:22:13.253344 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:13.253363 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:13.253428 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:13.253558 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:13.253559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:13.253724 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:13.253775 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.253777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.254433 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.254434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.255000 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.255002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.255422 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.255423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.255468 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.256006 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:13.256012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:13.257922 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
W20260619 10:22:13.258153 28869 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:13.253291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:60635 (local address 127.19.65.190:39053)
0619 10:22:13.253359 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:13.253361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:13.253410 (+ 49us) server_negotiation.cc:407] Connection header received
0619 10:22:13.253474 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:13.253475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:13.253485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:13.253530 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:13.253870 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.254347 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.255109 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.255283 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.255407 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.258069 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:60635: BlockingRecv error: recv got EOF from 127.19.65.131:60635 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:13 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:13.986514 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:13.977601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:44691)
0619 10:22:13.977747 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:13.977776 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:13.977836 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:13.977980 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:13.977980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:13.978142 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:13.978191 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.978193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.978800 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.978801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.979401 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:13.979403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.979754 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.979755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:13.979804 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.984715 (+ 4911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:13.984720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:13.986457 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
W20260619 10:22:13.986660 28873 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:13.977695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44691 (local address 127.19.65.190:39053)
0619 10:22:13.977824 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:13.977827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:13.977836 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:22:13.977890 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:13.977891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:13.977900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:13.977964 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:13.978255 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.978737 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.979469 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:13.979669 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:13.979733 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:13.986542 (+ 6809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44691: BlockingRecv error: recv got EOF from 127.19.65.129:44691 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:14.109808 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:14.105283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:58991)
0619 10:22:14.105418 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:14.105428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:14.105488 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:14.105657 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:14.105657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:14.105821 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:14.105873 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.105874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.106520 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.106521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.107038 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.107040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.107379 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.107380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.107420 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.107851 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:14.107860 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:14.109729 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260619 10:22:14.109959 28873 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:14.105368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:58991 (local address 127.19.65.190:39053)
0619 10:22:14.105416 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:14.105423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:14.105486 (+ 63us) server_negotiation.cc:407] Connection header received
0619 10:22:14.105574 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:14.105574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:14.105584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:14.105633 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:14.105973 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.106443 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.107123 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.107324 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.107376 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.109859 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:58991: BlockingRecv error: recv got EOF from 127.19.65.130:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:14.263240 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:14.258434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:57413)
0619 10:22:14.258568 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:14.258579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:14.258633 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:14.258779 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:14.258780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:14.258934 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:14.258994 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.258995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.259679 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.259680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.260253 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.260254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.260570 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.260571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.260619 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.261090 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:14.261095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:14.263129 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
W20260619 10:22:14.263366 28873 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:14.258523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:57413 (local address 127.19.65.190:39053)
0619 10:22:14.258594 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:14.258596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:14.258621 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:22:14.258691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:14.258691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:14.258705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:14.258764 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:14.259107 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.259607 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.260326 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.260502 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.260553 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.263246 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:57413: BlockingRecv error: recv got EOF from 127.19.65.131:57413 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:14 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:14.991714 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:14.986962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:46795)
0619 10:22:14.987122 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:14.987136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:14.987184 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:14.987418 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:14.987419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:14.987572 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:14.987617 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.987618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.988326 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.988327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.988850 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:14.988852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.989262 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.989263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:14.989328 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.989835 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:14.989839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:14.991610 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
W20260619 10:22:14.991758 28877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:14.987140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:46795 (local address 127.19.65.190:39053)
0619 10:22:14.987265 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:14.987267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:14.987274 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:22:14.987330 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:14.987331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:14.987340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:14.987400 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:14.987698 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.988235 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.988969 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:14.989171 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:14.989259 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:14.991687 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:46795: BlockingRecv error: recv got EOF from 127.19.65.129:46795 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:15.115186 28878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:15.110239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:47951)
0619 10:22:15.110427 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:15.110439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:15.110524 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:15.110689 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:15.110690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:15.110890 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:15.110934 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.110936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.111692 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.111693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.112222 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.112224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.112568 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.112569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.112625 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.113045 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:15.113050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:15.115131 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
W20260619 10:22:15.115340 28877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:15.110335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:47951 (local address 127.19.65.190:39053)
0619 10:22:15.110412 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:15.110414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:15.110507 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:22:15.110599 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:15.110600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:15.110612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:15.110683 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:15.111014 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.111605 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.112307 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.112493 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.112570 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.115243 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:47951: BlockingRecv error: recv got EOF from 127.19.65.130:47951 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:15.268287 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:15.263655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:44193)
0619 10:22:15.263773 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:15.263789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:15.263863 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:15.264000 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:15.264001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:15.264191 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:15.264236 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.264238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.264880 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.264881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.265398 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.265400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.265753 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.265754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.265821 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.266234 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:15.266239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:15.268230 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
W20260619 10:22:15.268426 28877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:15.263783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:44193 (local address 127.19.65.190:39053)
0619 10:22:15.263843 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:15.263845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:15.263853 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:22:15.263916 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:15.263917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:15.263932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:15.263994 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:15.264317 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.264798 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.265485 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.265672 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.265751 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.268347 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:44193: BlockingRecv error: recv got EOF from 127.19.65.131:44193 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:15 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:15.997099 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:15.992176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:44535)
0619 10:22:15.992353 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:15.992364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:15.992429 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:15.992581 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:15.992582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:15.992740 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:15.992822 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.992824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.993586 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.993588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.994183 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:15.994185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.994508 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.994508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:15.994560 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.995060 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:15.995066 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:15.997005 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
W20260619 10:22:15.997222 28881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:15.992261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44535 (local address 127.19.65.190:39053)
0619 10:22:15.992363 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:15.992365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:15.992434 (+ 69us) server_negotiation.cc:407] Connection header received
0619 10:22:15.992485 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:15.992486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:15.992498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:15.992576 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:15.992956 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.993476 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.994254 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:15.994444 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:15.994499 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:15.997118 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:44535: BlockingRecv error: recv got EOF from 127.19.65.129:44535 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:16.120363 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:16.115767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:51717)
0619 10:22:16.115878 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:16.115896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:16.115942 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:16.116070 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:16.116071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:16.116231 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:16.116274 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.116276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.116915 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.116916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.117568 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.117572 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.118011 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.118012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.118068 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:16.118501 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:16.118505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:16.120255 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260619 10:22:16.120406 28881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:16.115837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51717 (local address 127.19.65.190:39053)
0619 10:22:16.115929 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:16.115931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:16.115941 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:16.115991 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:16.115992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:16.116002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:16.116066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:16.116338 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.116826 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.117654 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.117938 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.118013 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:16.120345 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:51717: BlockingRecv error: recv got EOF from 127.19.65.130:51717 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:16.273329 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:16.268749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:58895)
0619 10:22:16.268862 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:16.268875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:16.268943 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:16.269121 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:16.269122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:16.269320 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:16.269362 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.269364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.270006 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.270007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.270538 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.270539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.270886 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.270887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.270945 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:16.271485 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:16.271490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:16.273262 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
W20260619 10:22:16.273474 28881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:16.268816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:58895 (local address 127.19.65.190:39053)
0619 10:22:16.268864 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:16.268866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:16.268924 (+ 58us) server_negotiation.cc:407] Connection header received
0619 10:22:16.268997 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:16.268998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:16.269012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:16.269107 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:16.269442 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.269918 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.270630 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.270814 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.270888 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:16.273373 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:58895: BlockingRecv error: recv got EOF from 127.19.65.131:58895 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:16 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:17.002043 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:16.997534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:53067)
0619 10:22:16.997669 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:16.997686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:16.997731 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:16.997868 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:16.997869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:16.998017 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:16.998060 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.998061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.998702 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.998702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.999241 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:16.999243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.999633 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.999633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:16.999676 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.000107 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:17.000112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:17.001961 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260619 10:22:17.002156 28885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:16.997611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53067 (local address 127.19.65.190:39053)
0619 10:22:16.997726 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:16.997728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:16.997761 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:22:16.997781 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:16.997781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:16.997791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:16.997849 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:16.998157 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.998638 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.999371 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:16.999566 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:16.999618 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.002058 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:53067: BlockingRecv error: recv got EOF from 127.19.65.129:53067 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
Jun 19 10:22:17 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:17 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:17.125613 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:17.120858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:44345)
0619 10:22:17.121050 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:17.121063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:17.121173 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:17.121357 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:17.121358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:17.121555 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:17.121600 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:17.121602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.122295 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.122296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:17.122824 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:17.122826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.123191 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.123192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:17.123244 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.123689 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:17.123693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:17.125546 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260619 10:22:17.125720 28885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:17.120953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:44345 (local address 127.19.65.190:39053)
0619 10:22:17.121047 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:17.121049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:17.121151 (+ 102us) server_negotiation.cc:407] Connection header received
0619 10:22:17.121250 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:17.121251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:17.121265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:17.121340 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:17.121683 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.122222 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.122911 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.123109 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.123190 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.125634 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:44345: BlockingRecv error: recv got EOF from 127.19.65.130:44345 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 10:22:17 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:17 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:17.278935 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:17.273888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:33019)
0619 10:22:17.274019 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:17.274033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:17.274107 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:17.274295 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:17.274296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:17.274494 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:17.274540 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:17.274542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.275318 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.275319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:17.275852 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:17.275854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.276328 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.276329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:17.276391 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.276841 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:17.276846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:17.278833 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
W20260619 10:22:17.278978 28885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:17.273969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:33019 (local address 127.19.65.190:39053)
0619 10:22:17.274048 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:17.274051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:17.274086 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:22:17.274176 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:17.274177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:17.274192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:17.274271 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:17.274634 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.275181 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.275943 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:17.276247 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:17.276327 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:17.278914 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:33019: BlockingRecv error: recv got EOF from 127.19.65.131:33019 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:18.007100 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:18.002539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:36395)
0619 10:22:18.002719 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:18.002728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:18.002797 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:18.002925 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:18.002925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:18.003118 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:18.003169 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.003171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.003851 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.003852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.004389 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.004391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.004734 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.004735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.004773 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.005242 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:18.005246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:18.007034 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
W20260619 10:22:18.007172 28889 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:18.002614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36395 (local address 127.19.65.190:39053)
0619 10:22:18.002718 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:18.002720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:18.002767 (+ 47us) server_negotiation.cc:407] Connection header received
0619 10:22:18.002837 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:18.002837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:18.002847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:18.002900 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:18.003251 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.003773 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.004467 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.004668 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.004721 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.007101 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:36395: BlockingRecv error: recv got EOF from 127.19.65.129:36395 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:18.130736 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:18.126053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:50737)
0619 10:22:18.126187 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:18.126200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:18.126264 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:18.126409 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:18.126411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:18.126591 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:18.126636 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.126638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.127327 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.127328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.127838 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.127839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.128198 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.128199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.128279 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.128759 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:18.128765 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:18.130626 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
W20260619 10:22:18.130782 28889 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:18.126166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50737 (local address 127.19.65.190:39053)
0619 10:22:18.126221 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:18.126223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:18.126246 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:22:18.126319 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:18.126319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:18.126331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:18.126403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:18.126713 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.127208 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.127916 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.128115 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.128194 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.130720 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:50737: BlockingRecv error: recv got EOF from 127.19.65.130:50737 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:18 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:18.284245 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:18.279428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:45837)
0619 10:22:18.279570 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:18.279581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:18.279651 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:18.279799 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:18.279800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:18.280035 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:18.280119 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.280121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.280811 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.280811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.281347 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:18.281348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.281851 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.281852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:18.281897 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.282345 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:18.282349 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:18.284176 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260619 10:22:18.284312 28889 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:18.279492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:45837 (local address 127.19.65.190:39053)
0619 10:22:18.279543 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:18.279545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:18.279640 (+ 95us) server_negotiation.cc:407] Connection header received
0619 10:22:18.279692 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:18.279693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:18.279705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:18.279756 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:18.280198 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.280749 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.281507 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:18.281777 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:18.281842 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:18.284266 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:45837: BlockingRecv error: recv got EOF from 127.19.65.131:45837 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:19.015735 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:19.007642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:45761)
0619 10:22:19.007840 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:19.007851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:19.007908 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:19.008050 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:19.008051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:19.008208 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:19.008254 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.008255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.008866 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.008866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.009382 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.009384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.009702 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.009703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.009744 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.013691 (+ 3947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:19.013696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:19.015677 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260619 10:22:19.015868 28893 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:19.007763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:45761 (local address 127.19.65.190:39053)
0619 10:22:19.007919 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:19.007920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:19.007930 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:19.007952 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:19.007953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:19.007963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:19.008033 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:19.008322 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.008800 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.009453 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.009635 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.009684 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.015779 (+ 6095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:45761: BlockingRecv error: recv got EOF from 127.19.65.129:45761 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:19.135898 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:19.131406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:41625)
0619 10:22:19.131611 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:19.131622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:19.131687 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:19.131878 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:19.131879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:19.132057 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:19.132100 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.132101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.132745 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.132746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.133235 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.133236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.133558 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.133559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.133607 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.134081 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:19.134085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:19.135840 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
W20260619 10:22:19.136052 28893 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:19.131500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41625 (local address 127.19.65.190:39053)
0619 10:22:19.131594 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:19.131596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:19.131669 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:22:19.131742 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:19.131743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:19.131755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:19.131843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:19.132179 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.132661 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.133312 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.133494 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.133560 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.135960 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:41625: BlockingRecv error: recv got EOF from 127.19.65.130:41625 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:19 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.131@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:19.289784 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:19.284710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.131:59691)
0619 10:22:19.284905 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:19.284917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:19.285020 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:19.285198 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:19.285199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:19.285392 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:19.285439 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.285441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.286188 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.286189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.286737 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:19.286738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.287154 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.287155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:19.287217 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.287793 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:19.287799 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:19.289726 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
W20260619 10:22:19.289904 28893 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:19.284806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59691 (local address 127.19.65.190:39053)
0619 10:22:19.284878 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:19.284880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:19.285024 (+ 144us) server_negotiation.cc:407] Connection header received
0619 10:22:19.285096 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:19.285097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:19.285115 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:19.285192 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:19.285543 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.286091 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.286862 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:19.287062 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:19.287154 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:19.289826 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.131:59691: BlockingRecv error: recv got EOF from 127.19.65.131:59691 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 10:22:20 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:20 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.129@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:20.020946 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.016322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.129:50465)
0619 10:22:20.016475 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:20.016487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:20.016543 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:20.016685 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:20.016686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:20.016835 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:20.016883 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.016885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.017582 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.017582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.018120 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.018122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.018460 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.018461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.018499 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.019002 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:20.019007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:20.020876 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260619 10:22:20.021162 28897 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:20.016408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:50465 (local address 127.19.65.190:39053)
0619 10:22:20.016540 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.016542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.016552 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:20.016594 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.016594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.016605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.016658 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.016966 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.017509 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.018200 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.018394 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.018447 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.020978 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.129:50465: BlockingRecv error: recv got EOF from 127.19.65.129:50465 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
Jun 19 10:22:20 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:22:20 dist-test-slave-xd2q krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.65.130@KRBTEST.COM for oryx/127.19.65.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:22:20.140733 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.136408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:39053 (local address 127.19.65.130:51139)
0619 10:22:20.136532 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:20.136548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:20.136601 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:20.136736 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:20.136737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:20.136901 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:20.136946 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.136947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.137569 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.137569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.138098 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.138100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.138445 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.138446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.138483 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.138926 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:20.138931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:20.140668 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.65.190:39053: Server oryx/127.19.65.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260619 10:22:20.140837 28897 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:20.136481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51139 (local address 127.19.65.190:39053)
0619 10:22:20.136579 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.136581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.136591 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:20.136661 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.136661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.136673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.136718 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.137022 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.137504 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.138172 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.138361 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.138440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.140766 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.65.130:51139: BlockingRecv error: recv got EOF from 127.19.65.130:51139 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 10:22:20.209056 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28420
I20260619 10:22:20.213812 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28572
I20260619 10:22:20.217865 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28707
I20260619 10:22:20.221706 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28350
2026-06-19T10:22:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17378 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T10:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:20Z Disabled control of system clock
I20260619 10:22:20.249157 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45487
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 10:22:20.327518 28902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:20.327718 28902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:20.327736 28902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:20.328897 28902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:20.328948 28902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:20.328959 28902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:20.328975 28902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:20.328987 28902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:20.330257 28902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:45487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:20.330451 28902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:20.330696 28902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:22:20.335794 28902 server_base.cc:1034] running on GCE node
W20260619 10:22:20.335754 28908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.335760 28910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.335767 28907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:20.336238 28902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:20.336514 28902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:20.337662 28902 hybrid_clock.cc:648] HybridClock initialized: now 1781864540337649 us; error 30 us; skew 500 ppm
I20260619 10:22:20.338881 28902 webserver.cc:460] Webserver started at http://127.19.65.190:42283/ using document root <none> and password file <none>
I20260619 10:22:20.339123 28902 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:20.339181 28902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:20.339370 28902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:20.340384 28902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "76252421cf6449b5bd5bdc6bb8b44419"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "426e222e5d8f6b977ff08909edd20dfa"
server_key_iv: "aa64e5205dfa2cffd9b1ab03a85f684c"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.340728 28902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "76252421cf6449b5bd5bdc6bb8b44419"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "426e222e5d8f6b977ff08909edd20dfa"
server_key_iv: "aa64e5205dfa2cffd9b1ab03a85f684c"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.341964 28902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:20.342674 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.342869 28902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:20.342965 28902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "76252421cf6449b5bd5bdc6bb8b44419"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "426e222e5d8f6b977ff08909edd20dfa"
server_key_iv: "aa64e5205dfa2cffd9b1ab03a85f684c"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.343045 28902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:20.364181 28902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:20.364468 28902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:20.364629 28902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:20.368549 28902 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:45487
I20260619 10:22:20.368584 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:45487 every 8 connection(s)
I20260619 10:22:20.368906 28902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:20.369522 28969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:20.371932 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419: Bootstrap starting.
I20260619 10:22:20.372411 28969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:20.372641 28969 log.cc:824] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:20.373114 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28902
I20260619 10:22:20.373221 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:22:20.373381 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419: No bootstrap required, opened a new log
I20260619 10:22:20.373425 19718 external_mini_cluster.cc:1428] Setting key 6844080477a541bd55daa323c7f827d0
I20260619 10:22:20.375720 28969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } }
I20260619 10:22:20.375888 28969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:20.375931 28969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76252421cf6449b5bd5bdc6bb8b44419, State: Initialized, Role: FOLLOWER
I20260619 10:22:20.376087 28969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } }
I20260619 10:22:20.376187 28969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:20.376227 28969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:20.376284 28969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:20.376895 28969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } }
I20260619 10:22:20.376999 28969 leader_election.cc:302] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76252421cf6449b5bd5bdc6bb8b44419; no voters:
I20260619 10:22:20.377147 28969 leader_election.cc:288] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:22:20.377187 28974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:20.377347 28974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [term 1 LEADER]: Becoming Leader. State: Replica: 76252421cf6449b5bd5bdc6bb8b44419, State: Running, Role: LEADER
I20260619 10:22:20.377370 28969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:20.378609 28974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } }
I20260619 10:22:20.379123 28976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76252421cf6449b5bd5bdc6bb8b44419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } } }
I20260619 10:22:20.379217 28976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:20.379438 28975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76252421cf6449b5bd5bdc6bb8b44419. Latest consensus state: current_term: 1 leader_uuid: "76252421cf6449b5bd5bdc6bb8b44419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76252421cf6449b5bd5bdc6bb8b44419" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 45487 } } }
I20260619 10:22:20.379623 28988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:20.379757 28975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:20.380226 28988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:22:20.380256 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.373890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.19.65.190:45487)
0619 10:22:20.374098 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.374107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.374140 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:22:20.374776 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.374785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.374857 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.375013 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.375148 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.377021 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.378206 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.378694 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.378806 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.379530 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:20.379534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:20.379534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:20.379768 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:20.379808 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:20.379823 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:20.379890 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:20.379975 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 10:22:20.380049 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260619 10:22:20.382012 28988 catalog_manager.cc:1347] Generated new cluster ID: 70ffaece12944f98b82e67636898cd98
I20260619 10:22:20.382063 28988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:22:20.410328 28988 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:22:20.411002 28988 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:20.414813 28988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76252421cf6449b5bd5bdc6bb8b44419: Generated new TSK 0
I20260619 10:22:20.415028 28988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:22:20.424405 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--builtin_ntp_servers=127.19.65.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 10:22:20.503340 28993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:20.503540 28993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:20.503561 28993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:20.504719 28993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:20.504757 28993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:20.504782 28993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:20.506068 28993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:20.506285 28993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:20.506577 28993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:20.511282 29001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.511317 28998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.511308 28999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:20.511507 28993 server_base.cc:1034] running on GCE node
I20260619 10:22:20.511811 28993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:20.512007 28993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:20.513144 28993 hybrid_clock.cc:648] HybridClock initialized: now 1781864540513119 us; error 41 us; skew 500 ppm
I20260619 10:22:20.514245 28993 webserver.cc:460] Webserver started at http://127.19.65.129:46073/ using document root <none> and password file <none>
I20260619 10:22:20.514477 28993 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:20.514520 28993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:20.514672 28993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:20.515772 28993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "0f43aaeaddf14476aa491262e90e136c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "6ec7de12512321a364ead3cefdad131f"
server_key_iv: "3ab59781e7ffc9f347429850060ce70b"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.516113 28993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "0f43aaeaddf14476aa491262e90e136c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "6ec7de12512321a364ead3cefdad131f"
server_key_iv: "3ab59781e7ffc9f347429850060ce70b"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.517429 28993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 10:22:20.518133 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.518291 28993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:20.518389 28993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "0f43aaeaddf14476aa491262e90e136c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "6ec7de12512321a364ead3cefdad131f"
server_key_iv: "3ab59781e7ffc9f347429850060ce70b"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.518477 28993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:20.550014 28993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:20.550292 28993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:20.550433 28993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:20.550683 28993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:20.551007 28993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:20.551059 28993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.551093 28993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:20.551144 28993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.556949 28993 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:33577
I20260619 10:22:20.556988 29120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:33577 every 8 connection(s)
I20260619 10:22:20.557336 28993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:22:20.558984 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 28993
I20260619 10:22:20.559075 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:22:20.559334 19718 external_mini_cluster.cc:1428] Setting key 44edf4387b090b894ec0f9e4d7873935
I20260619 10:22:20.560248 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--builtin_ntp_servers=127.19.65.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 10:22:20.562719 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.558069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:48911 (local address 127.19.65.190:45487)
0619 10:22:20.558145 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.558148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.558325 (+ 177us) server_negotiation.cc:407] Connection header received
0619 10:22:20.558769 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.558770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.558784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.558829 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.559777 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.560312 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.561303 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.561511 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.561593 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.562504 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:20.562505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:20.562505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:20.562509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:20.562523 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:20.562526 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:20.562570 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:20.562645 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 10:22:20.562665 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 10:22:20.563004 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.558029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45487 (local address 127.19.65.129:48911)
0619 10:22:20.558214 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:20.558244 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:20.558694 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:20.558899 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:20.558903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:20.558914 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:20.559679 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.559686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.560428 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.560429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.561211 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.561214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.561603 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.561604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.561755 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.562341 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:20.562342 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:20.562378 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:20.562404 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:20.562572 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:20.562575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:20.562578 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:20.562637 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:22:20.562736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260619 10:22:20.563537 29121 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45487
I20260619 10:22:20.563619 29121 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:20.563803 29121 heartbeater.cc:511] Master 127.19.65.190:45487 requested a full tablet report, sending...
I20260619 10:22:20.564291 28933 ts_manager.cc:194] Registered new tserver with Master: 0f43aaeaddf14476aa491262e90e136c (127.19.65.129:33577)
I20260619 10:22:20.565311 28933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.129:48911
W20260619 10:22:20.638800 29124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:20.638978 29124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:20.639005 29124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:20.640130 29124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:20.640168 29124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:20.640192 29124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:20.641398 29124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:20.641569 29124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:20.641785 29124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:20.646708 29129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.646708 29130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.646901 29132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:20.646847 29124 server_base.cc:1034] running on GCE node
I20260619 10:22:20.647161 29124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:20.647421 29124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:20.648598 29124 hybrid_clock.cc:648] HybridClock initialized: now 1781864540648594 us; error 51 us; skew 500 ppm
I20260619 10:22:20.649893 29124 webserver.cc:460] Webserver started at http://127.19.65.130:41261/ using document root <none> and password file <none>
I20260619 10:22:20.650137 29124 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:20.650208 29124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:20.650354 29124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:20.651389 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "dc037dfd122a49ca863281fd3e8e1478"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "c6d82ac93ec64ffeb39a623ce475ad3a"
server_key_iv: "77857102f6b3f5904bfe55a5a4e914bc"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.651741 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "dc037dfd122a49ca863281fd3e8e1478"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "c6d82ac93ec64ffeb39a623ce475ad3a"
server_key_iv: "77857102f6b3f5904bfe55a5a4e914bc"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.653048 29124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:20.653721 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.653888 29124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.653978 29124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "dc037dfd122a49ca863281fd3e8e1478"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "c6d82ac93ec64ffeb39a623ce475ad3a"
server_key_iv: "77857102f6b3f5904bfe55a5a4e914bc"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.654064 29124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:20.662050 29124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:20.662305 29124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:20.662447 29124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:20.662657 29124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:20.662955 29124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:20.663010 29124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.663060 29124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:20.663091 29124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.668668 29124 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:40623
I20260619 10:22:20.668715 29251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:40623 every 8 connection(s)
I20260619 10:22:20.668995 29124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:22:20.674001 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.669685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50309 (local address 127.19.65.190:45487)
0619 10:22:20.669765 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.669768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.669952 (+ 184us) server_negotiation.cc:407] Connection header received
0619 10:22:20.670436 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.670437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.670448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.670499 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.671187 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.671706 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.672660 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.672851 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.672922 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.673716 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:20.673717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:20.673718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:20.673722 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:20.673737 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:20.673742 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:20.673808 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:20.673900 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 10:22:20.673925 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:22:20.674197 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.669645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45487 (local address 127.19.65.130:50309)
0619 10:22:20.669841 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:20.669875 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:20.670339 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:20.670590 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:20.670598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:20.670605 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:20.671096 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.671100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.671798 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.671799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.672559 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.672561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.672940 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.672941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.673065 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.673568 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:20.673568 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:20.673594 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:20.673614 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:20.673808 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:20.673814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:20.673817 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:20.673903 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 10:22:20.673995 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260619 10:22:20.674631 29252 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45487
I20260619 10:22:20.674736 29252 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:20.674911 29252 heartbeater.cc:511] Master 127.19.65.190:45487 requested a full tablet report, sending...
I20260619 10:22:20.675209 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29124
I20260619 10:22:20.675343 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:22:20.675364 28933 ts_manager.cc:194] Registered new tserver with Master: dc037dfd122a49ca863281fd3e8e1478 (127.19.65.130:40623)
I20260619 10:22:20.675561 19718 external_mini_cluster.cc:1428] Setting key ecf200e314ec65d499b04816ce5f8710
I20260619 10:22:20.675820 28933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.130:50309
I20260619 10:22:20.676608 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--builtin_ntp_servers=127.19.65.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 10:22:20.758764 29255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:20.758942 29255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:20.758970 29255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:20.760115 29255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:20.760154 29255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:20.760177 29255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:20.761502 29255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:20.761664 29255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:20.761873 29255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:20.766386 29261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.766386 29263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.766433 29260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:20.766571 29255 server_base.cc:1034] running on GCE node
I20260619 10:22:20.766826 29255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:20.767073 29255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:20.768234 29255 hybrid_clock.cc:648] HybridClock initialized: now 1781864540768243 us; error 50 us; skew 500 ppm
I20260619 10:22:20.769574 29255 webserver.cc:460] Webserver started at http://127.19.65.131:39017/ using document root <none> and password file <none>
I20260619 10:22:20.769796 29255 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:20.769863 29255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:20.770004 29255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:20.770974 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "bcb40bb2c1bc4807b5591698236b463c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "3f127aec47c129c6e5b15f4e449e6d38"
server_key_iv: "a92cc8899be9eaf491262c300b319193"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.771407 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "bcb40bb2c1bc4807b5591698236b463c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "3f127aec47c129c6e5b15f4e449e6d38"
server_key_iv: "a92cc8899be9eaf491262c300b319193"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.772665 29255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:20.773461 29269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.773643 29255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:20.773737 29255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "bcb40bb2c1bc4807b5591698236b463c"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "3f127aec47c129c6e5b15f4e449e6d38"
server_key_iv: "a92cc8899be9eaf491262c300b319193"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.773823 29255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:20.789644 29255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:20.789901 29255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:20.790045 29255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:20.790266 29255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:20.790560 29255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:20.790614 29255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.790652 29255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:20.790681 29255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.796175 29255 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38055
I20260619 10:22:20.796217 29382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:38055 every 8 connection(s)
I20260619 10:22:20.796509 29255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:22:20.801167 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.797158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:39389 (local address 127.19.65.190:45487)
0619 10:22:20.797221 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.797224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.797438 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:22:20.797873 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.797874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.797884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.797935 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.798539 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.799021 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.799927 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.800121 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.800174 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.800856 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:20.800857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:20.800857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:20.800861 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:20.800876 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:20.800879 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:20.800953 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:20.801047 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 10:22:20.801074 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:22:20.801153 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29255
I20260619 10:22:20.801247 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:22:20.801260 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.797159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:45487 (local address 127.19.65.131:39389)
0619 10:22:20.797326 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:20.797355 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:20.797797 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:20.797985 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:20.797989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:20.797999 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:20.798452 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.798456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.799093 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.799094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.799856 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:20.799858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:20.800190 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:20.800191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:20.800290 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:20.800734 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:20.800734 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:20.800759 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:20.800778 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:20.800952 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:20.800954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:20.800956 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:22:20.801018 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 10:22:20.801088 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260619 10:22:20.801456 19718 external_mini_cluster.cc:1428] Setting key 153850c66deb03eccf9b75646eb44712
I20260619 10:22:20.801761 29383 heartbeater.cc:347] Connected to a master server at 127.19.65.190:45487
I20260619 10:22:20.801875 29383 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:20.802079 29383 heartbeater.cc:511] Master 127.19.65.190:45487 requested a full tablet report, sending...
I20260619 10:22:20.802448 28932 ts_manager.cc:194] Registered new tserver with Master: bcb40bb2c1bc4807b5591698236b463c (127.19.65.131:38055)
I20260619 10:22:20.802872 28932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.131:39389
I20260619 10:22:20.813150 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:22:20.814967 28973 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:20.814444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.19.65.190:45487)
0619 10:22:20.814519 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.814521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.814610 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:22:20.814684 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.814685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.814694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.814752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.814889 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55410: BlockingRecv error: recv got EOF from 127.0.0.1:55410 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:22:20.815091 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28993
I20260619 10:22:20.819329 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29124
I20260619 10:22:20.823397 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29255
I20260619 10:22:20.827284 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 28902
2026-06-19T10:22:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (604 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T10:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:20Z Disabled control of system clock
I20260619 10:22:20.852718 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43221
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 10:22:20.931103 29394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:20.931278 29394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:20.931316 29394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:20.932384 29394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:20.932415 29394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:20.932428 29394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:20.932440 29394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:20.932452 29394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:20.933647 29394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:20.933835 29394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:20.934049 29394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:20.938505 29402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.938498 29399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:20.938519 29400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:20.938521 29394 server_base.cc:1034] running on GCE node
I20260619 10:22:20.938906 29394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:20.939217 29394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:20.940392 29394 hybrid_clock.cc:648] HybridClock initialized: now 1781864540940386 us; error 55 us; skew 500 ppm
I20260619 10:22:20.941566 29394 webserver.cc:460] Webserver started at http://127.19.65.190:33025/ using document root <none> and password file <none>
I20260619 10:22:20.941797 29394 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:20.941861 29394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:20.941985 29394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:20.942901 29394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "2bf7f243c80b46558a9c1d2748acaf73"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "b10db33533287f06e5eec8ea7237506f"
server_key_iv: "28fc3ea106324b9a38878df4b5e4a0d7"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.943264 29394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "2bf7f243c80b46558a9c1d2748acaf73"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "b10db33533287f06e5eec8ea7237506f"
server_key_iv: "28fc3ea106324b9a38878df4b5e4a0d7"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.944430 29394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:20.945027 29408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:20.945155 29394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:20.945240 29394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "2bf7f243c80b46558a9c1d2748acaf73"
format_stamp: "Formatted at 2026-06-19 10:22:20 on dist-test-slave-xd2q"
server_key: "b10db33533287f06e5eec8ea7237506f"
server_key_iv: "28fc3ea106324b9a38878df4b5e4a0d7"
server_key_version: "encryptionkey@0"
I20260619 10:22:20.945309 29394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:20.948313 29394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:20.948588 29394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:20.948722 29394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:20.952354 29394 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43221
I20260619 10:22:20.952380 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:43221 every 8 connection(s)
I20260619 10:22:20.952665 29394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:20.953240 29461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:20.955762 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73: Bootstrap starting.
I20260619 10:22:20.956192 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29394
I20260619 10:22:20.956298 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:22:20.956337 29461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:20.956501 19718 external_mini_cluster.cc:1428] Setting key 9b27991f1902552ccfc4e2c0581d7a45
I20260619 10:22:20.956717 29461 log.cc:824] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:20.957547 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73: No bootstrap required, opened a new log
I20260619 10:22:20.959178 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:20.957057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.19.65.190:43221)
0619 10:22:20.957291 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:20.957299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:20.957335 (+ 36us) server_negotiation.cc:407] Connection header received
0619 10:22:20.957837 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:20.957845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:20.957891 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:20.958077 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:20.958750 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:20.958753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:20.958754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:20.958758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:20.958802 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:20.958816 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:20.958907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:20.958938 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 10:22:20.958998 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260619 10:22:20.959481 29461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } }
I20260619 10:22:20.959620 29461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:20.959676 29461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf7f243c80b46558a9c1d2748acaf73, State: Initialized, Role: FOLLOWER
I20260619 10:22:20.959813 29461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } }
I20260619 10:22:20.959910 29461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:20.959939 29461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:20.959975 29461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:20.960695 29461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } }
I20260619 10:22:20.960794 29461 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf7f243c80b46558a9c1d2748acaf73; no voters:
I20260619 10:22:20.960951 29461 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:22:20.961032 29466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:20.961186 29461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:20.961202 29466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf7f243c80b46558a9c1d2748acaf73, State: Running, Role: LEADER
I20260619 10:22:20.961417 29466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } }
I20260619 10:22:20.961812 29468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf7f243c80b46558a9c1d2748acaf73. Latest consensus state: current_term: 1 leader_uuid: "2bf7f243c80b46558a9c1d2748acaf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } } }
I20260619 10:22:20.961973 29468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:20.961807 29467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf7f243c80b46558a9c1d2748acaf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf7f243c80b46558a9c1d2748acaf73" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43221 } } }
I20260619 10:22:20.962188 29475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:20.962172 29467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:20.962685 29475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:22:20.964331 29475 catalog_manager.cc:1347] Generated new cluster ID: 26467d78894f43b5be6ec62aeb23d326
I20260619 10:22:20.964401 29475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:22:20.976272 29475 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:22:20.976410 29475 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:20.978571 29475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf7f243c80b46558a9c1d2748acaf73: Generated new TSK 0
I20260619 10:22:20.978734 29475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:22:20.979790 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--builtin_ntp_servers=127.19.65.148:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 10:22:21.057192 29485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:21.057399 29485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:21.057435 29485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:21.058552 29485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:21.058614 29485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:21.058655 29485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:21.059978 29485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:21.060213 29485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:21.060478 29485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:21.065511 29493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:21.065552 29490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.065696 29485 server_base.cc:1034] running on GCE node
W20260619 10:22:21.065732 29491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.066061 29485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:21.066308 29485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:21.067503 29485 hybrid_clock.cc:648] HybridClock initialized: now 1781864541067490 us; error 31 us; skew 500 ppm
I20260619 10:22:21.068766 29485 webserver.cc:460] Webserver started at http://127.19.65.129:34889/ using document root <none> and password file <none>
I20260619 10:22:21.069010 29485 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:21.069088 29485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:21.069255 29485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:21.070250 29485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "230af88532154858a66af2e5fa56a175"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "1b8c90b77a0182fb922060681fffae29"
server_key_iv: "ae41be02addc2ccf6d98a9612bf6051f"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.070636 29485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "230af88532154858a66af2e5fa56a175"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "1b8c90b77a0182fb922060681fffae29"
server_key_iv: "ae41be02addc2ccf6d98a9612bf6051f"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.071997 29485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:22:21.072747 29499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.072914 29485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:22:21.072993 29485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "230af88532154858a66af2e5fa56a175"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "1b8c90b77a0182fb922060681fffae29"
server_key_iv: "ae41be02addc2ccf6d98a9612bf6051f"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.073045 29485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:21.076319 29485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:21.076565 29485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:21.076683 29485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:21.076884 29485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:21.077174 29485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:21.077224 29485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.077262 29485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:21.077287 29485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.083436 29485 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:34147
I20260619 10:22:21.083478 29612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:34147 every 8 connection(s)
I20260619 10:22:21.083765 29485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:22:21.084683 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29485
I20260619 10:22:21.084771 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:22:21.084970 19718 external_mini_cluster.cc:1428] Setting key 31a6ba9d502ba8d1b80a4a4235d58403
I20260619 10:22:21.086331 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--builtin_ntp_servers=127.19.65.148:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 10:22:21.086581 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.084438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53875 (local address 127.19.65.190:43221)
0619 10:22:21.084500 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:21.084502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:21.084691 (+ 189us) server_negotiation.cc:407] Connection header received
0619 10:22:21.085138 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:21.085139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:21.085145 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:21.085199 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:21.086318 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:21.086320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:21.086320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:21.086325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:21.086340 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:21.086344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:21.086389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:21.086501 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 10:22:21.086515 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:22:21.086786 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.084407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43221 (local address 127.19.65.129:53875)
0619 10:22:21.084561 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:21.084597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:21.085042 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:21.085262 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:21.085266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:21.085275 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:21.086165 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:21.086165 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:21.086193 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:21.086217 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:21.086397 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:21.086402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:21.086404 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:22:21.086508 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 10:22:21.086563 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260619 10:22:21.087195 29613 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43221
I20260619 10:22:21.087270 29613 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:21.087424 29613 heartbeater.cc:511] Master 127.19.65.190:43221 requested a full tablet report, sending...
I20260619 10:22:21.087888 29425 ts_manager.cc:194] Registered new tserver with Master: 230af88532154858a66af2e5fa56a175 (127.19.65.129:34147)
W20260619 10:22:21.165722 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:21.165917 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:21.165953 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:21.167075 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:21.167135 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:21.167176 29616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:21.168505 29616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:21.168747 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:21.169023 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:21.173817 29622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:21.173818 29621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.173897 29616 server_base.cc:1034] running on GCE node
W20260619 10:22:21.173839 29624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.174194 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:21.174388 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:21.175521 29616 hybrid_clock.cc:648] HybridClock initialized: now 1781864541175512 us; error 32 us; skew 500 ppm
I20260619 10:22:21.176832 29616 webserver.cc:460] Webserver started at http://127.19.65.130:34473/ using document root <none> and password file <none>
I20260619 10:22:21.177083 29616 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:21.177127 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:21.177284 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:21.178313 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "4a5239d0a8134f82a5c142060cd8d9e5"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "e81d2782367bcc4518fedeb974e8b02f"
server_key_iv: "ac9d0513a2ea75756463ac80ff508de8"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.178668 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "4a5239d0a8134f82a5c142060cd8d9e5"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "e81d2782367bcc4518fedeb974e8b02f"
server_key_iv: "ac9d0513a2ea75756463ac80ff508de8"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.179975 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:22:21.180672 29630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.180842 29616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:21.180927 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "4a5239d0a8134f82a5c142060cd8d9e5"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "e81d2782367bcc4518fedeb974e8b02f"
server_key_iv: "ac9d0513a2ea75756463ac80ff508de8"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.181001 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:21.184291 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:21.184532 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:21.184660 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:21.184866 29616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:21.185159 29616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:21.185191 29616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.185210 29616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:21.185256 29616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.190891 29616 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:33945
I20260619 10:22:21.190932 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:33945 every 8 connection(s)
I20260619 10:22:21.191223 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:22:21.191504 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29616
I20260619 10:22:21.191601 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:22:21.191779 19718 external_mini_cluster.cc:1428] Setting key c2370da81c51e66f32d4f4935ec29a05
I20260619 10:22:21.192771 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--builtin_ntp_servers=127.19.65.148:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 10:22:21.194518 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.191941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50577 (local address 127.19.65.190:43221)
0619 10:22:21.192007 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:21.192009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:21.192197 (+ 188us) server_negotiation.cc:407] Connection header received
0619 10:22:21.192897 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:21.192898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:21.192904 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:21.192951 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:21.194267 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:21.194268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:21.194269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:21.194273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:21.194287 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:21.194292 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:21.194343 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:21.194454 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 10:22:21.194467 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:22:21.194761 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.191902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43221 (local address 127.19.65.130:50577)
0619 10:22:21.192090 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:21.192125 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:21.192794 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:21.193031 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:21.193036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:21.193048 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:21.194092 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:21.194093 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:21.194136 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:21.194170 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:21.194371 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:21.194375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:21.194378 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:21.194451 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:22:21.194511 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260619 10:22:21.195385 29744 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43221
I20260619 10:22:21.195456 29744 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:21.195559 29744 heartbeater.cc:511] Master 127.19.65.190:43221 requested a full tablet report, sending...
I20260619 10:22:21.195894 29425 ts_manager.cc:194] Registered new tserver with Master: 4a5239d0a8134f82a5c142060cd8d9e5 (127.19.65.130:33945)
W20260619 10:22:21.271431 29747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:21.271639 29747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:21.271674 29747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:21.272773 29747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:21.272831 29747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:21.272869 29747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:21.274181 29747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:21.274432 29747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:21.274713 29747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:21.279488 29755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.279537 29747 server_base.cc:1034] running on GCE node
W20260619 10:22:21.279500 29752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:21.279508 29753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:21.279894 29747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:21.280151 29747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:21.281335 29747 hybrid_clock.cc:648] HybridClock initialized: now 1781864541281338 us; error 42 us; skew 500 ppm
I20260619 10:22:21.282567 29747 webserver.cc:460] Webserver started at http://127.19.65.131:44771/ using document root <none> and password file <none>
I20260619 10:22:21.282799 29747 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:21.282868 29747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:21.283023 29747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:21.284130 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "d619fb55ba514bdfb3d17577576cc985"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "8cb81d2a6f880f18f128a69210656cde"
server_key_iv: "97d7993451aaf91b434f8c1f238c8e0e"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.284528 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "d619fb55ba514bdfb3d17577576cc985"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "8cb81d2a6f880f18f128a69210656cde"
server_key_iv: "97d7993451aaf91b434f8c1f238c8e0e"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.285777 29747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:22:21.286469 29761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.286645 29747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:21.286736 29747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "d619fb55ba514bdfb3d17577576cc985"
format_stamp: "Formatted at 2026-06-19 10:22:21 on dist-test-slave-xd2q"
server_key: "8cb81d2a6f880f18f128a69210656cde"
server_key_iv: "97d7993451aaf91b434f8c1f238c8e0e"
server_key_version: "encryptionkey@0"
I20260619 10:22:21.286813 29747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:21.290169 29747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:21.290421 29747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:21.290560 29747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:21.290761 29747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:21.291066 29747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:21.291123 29747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.291164 29747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:21.291191 29747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:21.296904 29747 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45035
I20260619 10:22:21.296941 29874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45035 every 8 connection(s)
I20260619 10:22:21.297279 29747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:22:21.297890 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29747
I20260619 10:22:21.297999 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:22:21.298206 19718 external_mini_cluster.cc:1428] Setting key a692370045a22532db028cb83a4f46f4
I20260619 10:22:21.300562 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.298074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:41203 (local address 127.19.65.190:43221)
0619 10:22:21.298147 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:21.298149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:21.298381 (+ 232us) server_negotiation.cc:407] Connection header received
0619 10:22:21.299051 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:21.299052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:21.299059 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:21.299098 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:21.300331 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:21.300332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:21.300332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:22:21.300336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:21.300350 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:22:21.300355 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:21.300410 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:21.300502 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 10:22:21.300513 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:22:21.300748 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:21.298030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43221 (local address 127.19.65.131:41203)
0619 10:22:21.298262 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:21.298301 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:21.298936 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:21.299173 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:21.299177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:21.299187 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:21.300172 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:22:21.300173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:21.300199 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:22:21.300225 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:21.300423 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:21.300427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:21.300429 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:22:21.300493 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:22:21.300537 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260619 10:22:21.301229 29875 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43221
I20260619 10:22:21.301328 29875 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:21.301481 29875 heartbeater.cc:511] Master 127.19.65.190:43221 requested a full tablet report, sending...
I20260619 10:22:21.301820 29425 ts_manager.cc:194] Registered new tserver with Master: d619fb55ba514bdfb3d17577576cc985 (127.19.65.131:45035)
I20260619 10:22:21.309757 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:22:21.311534 29465 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:22:21.311021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.19.65.190:43221)
0619 10:22:21.311098 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:21.311100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:21.311200 (+ 100us) server_negotiation.cc:407] Connection header received
0619 10:22:21.311239 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:21.311240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:21.311244 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:21.311340 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:21.311472 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49904: BlockingRecv error: recv got EOF from 127.0.0.1:49904 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 10:22:21.311549 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29485
I20260619 10:22:21.315871 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29616
I20260619 10:22:21.319916 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29747
I20260619 10:22:21.323890 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29394
2026-06-19T10:22:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (496 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:22:21 dist-test-slave-xd2q krb5kdc[29885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:22:21 dist-test-slave-xd2q krb5kdc[29885](info): set up 2 sockets
Jun 19 10:22:21 dist-test-slave-xd2q krb5kdc[29885](info): commencing operation
krb5kdc: starting...
W20260619 10:22:22.340016 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:22:22.450402 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41803
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:41803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:22.532775 29898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:22.532975 29898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:22.532997 29898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:22.534158 29898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:22:22.534199 29898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:22.534214 29898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:22.534251 29898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:22:22.534277 29898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:22:22.535789 29898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:41803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:22.536037 29898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:22.536291 29898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:22.540910 29904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:22.540917 29903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:22.540966 29906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:22.541100 29898 server_base.cc:1034] running on GCE node
I20260619 10:22:22.541370 29898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:22.541673 29898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:22.542830 29898 hybrid_clock.cc:648] HybridClock initialized: now 1781864542542810 us; error 38 us; skew 500 ppm
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:22.545230 29898 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:22:22.545626 29898 webserver.cc:460] Webserver started at http://127.19.65.190:34761/ using document root <none> and password file <none>
I20260619 10:22:22.545838 29898 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:22.545902 29898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:22.546061 29898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:22.547019 29898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "1a5732244b8349568613b109c107c1e7"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "b76d9f0bc433be0c1ea83c4c26fcfe7a"
server_key_iv: "36478605ba0deec04280dc889a0ceced"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.547438 29898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "1a5732244b8349568613b109c107c1e7"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "b76d9f0bc433be0c1ea83c4c26fcfe7a"
server_key_iv: "36478605ba0deec04280dc889a0ceced"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.548715 29898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:22:22.549376 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.549548 29898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:22.549641 29898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "1a5732244b8349568613b109c107c1e7"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "b76d9f0bc433be0c1ea83c4c26fcfe7a"
server_key_iv: "36478605ba0deec04280dc889a0ceced"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.549736 29898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:22.557801 29898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:22.559922 29898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:22.560073 29898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:22.563649 29898 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:41803
I20260619 10:22:22.563680 29965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:41803 every 8 connection(s)
I20260619 10:22:22.563977 29898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:22:22.564254 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29898
I20260619 10:22:22.564361 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:22:22.564577 19718 external_mini_cluster.cc:1428] Setting key 9d47b521ee199426348216660cd6d450
I20260619 10:22:22.565088 29966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:22.567982 29966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: Bootstrap starting.
I20260619 10:22:22.568557 29966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:22.568861 29966 log.cc:824] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:22.569499 29966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: No bootstrap required, opened a new log
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:22.571345 29966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } }
I20260619 10:22:22.571516 29966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:22.571565 29966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5732244b8349568613b109c107c1e7, State: Initialized, Role: FOLLOWER
I20260619 10:22:22.571682 29966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } }
I20260619 10:22:22.571784 29966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:22:22.571826 29966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:22:22.571874 29966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:22.572527 29966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } }
I20260619 10:22:22.572633 29966 leader_election.cc:302] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a5732244b8349568613b109c107c1e7; no voters:
I20260619 10:22:22.572791 29966 leader_election.cc:288] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:22:22.572850 29971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:22.573024 29966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:22:22.573035 29971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a5732244b8349568613b109c107c1e7, State: Running, Role: LEADER
I20260619 10:22:22.573154 29971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } }
I20260619 10:22:22.573612 29971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a5732244b8349568613b109c107c1e7. Latest consensus state: current_term: 1 leader_uuid: "1a5732244b8349568613b109c107c1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } } }
I20260619 10:22:22.573621 29972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a5732244b8349568613b109c107c1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5732244b8349568613b109c107c1e7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41803 } } }
I20260619 10:22:22.573740 29971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:22.573799 29972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:22:22.573982 29980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:22:22.575037 29980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:22:22.575843 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:22.565069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.19.65.190:41803)
0619 10:22:22.565228 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:22.565236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:22.565262 (+ 26us) server_negotiation.cc:407] Connection header received
0619 10:22:22.565733 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:22.565741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:22.565797 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:22.565939 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:22.566556 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.567900 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.568766 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.569247 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.569373 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:22.571804 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:22.571808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:22.571809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:22.571821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:22.574008 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.574428 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.574454 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.574455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.574558 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.574832 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.574833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.574834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.575066 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:22.575240 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:22.575551 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 10:22:22.575686 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260619 10:22:22.576529 29980 catalog_manager.cc:1347] Generated new cluster ID: 478e1ef4c730471c8070918e6b66e003
I20260619 10:22:22.576576 29980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:22:22.590976 29980 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:22:22.591559 29980 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:22:22.595495 29980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: Generated new TSK 0
I20260619 10:22:22.595670 29980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:22:22.629571 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:22.721820 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:22.722043 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:22.722077 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:22.723178 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:22.723242 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:22.723285 29992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:22:22.724725 29992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:22.724968 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:22.725211 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:22:22.729979 29992 server_base.cc:1034] running on GCE node
W20260619 10:22:22.729951 29998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:22.729951 30000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:22.729947 29997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:22.730321 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:22.730553 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:22.731686 29992 hybrid_clock.cc:648] HybridClock initialized: now 1781864542731676 us; error 26 us; skew 500 ppm
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:22.734090 29992 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:22:22.734490 29992 webserver.cc:460] Webserver started at http://127.19.65.129:40951/ using document root <none> and password file <none>
I20260619 10:22:22.734709 29992 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:22.734771 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:22.734925 29992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:22.736019 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "51b68bc34826470aa945e9bedda85d02"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "2cc6d088145c2c0929b639e8775719db"
server_key_iv: "6db5214feded0b4a03a91f4933b811c3"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.736368 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "51b68bc34826470aa945e9bedda85d02"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "2cc6d088145c2c0929b639e8775719db"
server_key_iv: "6db5214feded0b4a03a91f4933b811c3"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.737680 29992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:22.738425 30007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.738612 29992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:22.738711 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "51b68bc34826470aa945e9bedda85d02"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "2cc6d088145c2c0929b639e8775719db"
server_key_iv: "6db5214feded0b4a03a91f4933b811c3"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.738790 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:22.750660 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:22.752627 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:22.752779 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:22.752981 29992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:22.753311 29992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:22.753373 29992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.753413 29992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:22.753451 29992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.759050 29992 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:35345
I20260619 10:22:22.759119 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:35345 every 8 connection(s)
I20260619 10:22:22.759483 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:22:22.764354 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 29992
I20260619 10:22:22.764469 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:22:22.764669 19718 external_mini_cluster.cc:1428] Setting key 06ecfaa23e760623039c13c25d7d33f1
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:22.769057 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:22.760160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:33837 (local address 127.19.65.190:41803)
0619 10:22:22.760258 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:22.760260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:22.760463 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:22:22.760949 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:22.760950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:22.760961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:22.761006 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:22.761906 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.762391 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.763395 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.763612 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.763695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:22.766003 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:22.766004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:22.766005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:22.766012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:22.767684 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.768055 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.768057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.768057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.768113 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.768362 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.768363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.768363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.768486 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:22.768582 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:22.768850 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 10:22:22.768935 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:22:22.769244 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:22.760126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41803 (local address 127.19.65.129:33837)
0619 10:22:22.760306 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:22.760370 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:22.760847 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:22.761084 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:22.761088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:22.761642 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:22.761807 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:22.761810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.762500 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.762502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:22.763252 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:22.763254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.763703 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.763705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:22.763823 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:22.764299 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:22.764309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:22.765889 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:22.767757 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:22.767762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:22.767765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:22.767977 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:22.768191 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:22.768192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:22.768192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:22.768295 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:22.768575 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:22.768581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:22.768717 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:22:22.768818 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:22:22.768986 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260619 10:22:22.769744 30121 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41803
I20260619 10:22:22.769833 30121 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:22.770025 30121 heartbeater.cc:511] Master 127.19.65.190:41803 requested a full tablet report, sending...
I20260619 10:22:22.770615 29930 ts_manager.cc:194] Registered new tserver with Master: 51b68bc34826470aa945e9bedda85d02 (127.19.65.129:35345)
I20260619 10:22:22.771598 29930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:33837
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:22:22.789803 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:22.870420 30126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:22.870594 30126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:22.870615 30126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:22.871693 30126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:22.871728 30126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:22.871754 30126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:22:22.873077 30126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:22.873327 30126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:22.873497 30126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:22.878315 30131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:22.878345 30132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:22.878508 30126 server_base.cc:1034] running on GCE node
W20260619 10:22:22.878338 30134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:22.878756 30126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:22.878991 30126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:22.880153 30126 hybrid_clock.cc:648] HybridClock initialized: now 1781864542880146 us; error 31 us; skew 500 ppm
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:22.882666 30126 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:22:22.883111 30126 webserver.cc:460] Webserver started at http://127.19.65.130:36907/ using document root <none> and password file <none>
I20260619 10:22:22.883416 30126 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:22.883515 30126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:22.883685 30126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:22.884693 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "7eabf0f07c7f4c43ae93b46c70594705"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "47834adc1e93cf750303a0ff66f5ca9f"
server_key_iv: "bc1bca5a5ae89d00342681109b6a8e3c"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.885076 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "7eabf0f07c7f4c43ae93b46c70594705"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "47834adc1e93cf750303a0ff66f5ca9f"
server_key_iv: "bc1bca5a5ae89d00342681109b6a8e3c"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.886464 30126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:22:22.887207 30141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.887385 30126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:22:22.887497 30126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "7eabf0f07c7f4c43ae93b46c70594705"
format_stamp: "Formatted at 2026-06-19 10:22:22 on dist-test-slave-xd2q"
server_key: "47834adc1e93cf750303a0ff66f5ca9f"
server_key_iv: "bc1bca5a5ae89d00342681109b6a8e3c"
server_key_version: "encryptionkey@0"
I20260619 10:22:22.887586 30126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:22.902415 30126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:22.904508 30126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:22.904670 30126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:22.904884 30126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:22.905187 30126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:22.905246 30126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.905287 30126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:22.905318 30126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:22.911484 30126 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:37337
I20260619 10:22:22.911517 30254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:37337 every 8 connection(s)
I20260619 10:22:22.911834 30126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:22:22.914346 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 30126
I20260619 10:22:22.914462 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:22:22.914674 19718 external_mini_cluster.cc:1428] Setting key 6da960f634b9e55f29298ad54cdfe0b5
Jun 19 10:22:22 dist-test-slave-xd2q krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864542, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:22.922622 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:22.912541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:58125 (local address 127.19.65.190:41803)
0619 10:22:22.912611 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:22.912613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:22.912769 (+ 156us) server_negotiation.cc:407] Connection header received
0619 10:22:22.913208 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:22.913209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:22.913219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:22.913294 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:22.914211 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.915013 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.916796 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.917160 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.917253 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:22.919413 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:22.919415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:22.919415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:22.919423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:22.921261 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.921663 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.921664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.921665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.921720 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:22.921986 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:22.921987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:22.921987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:22.922106 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:22.922181 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:22.922445 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 10:22:22.922536 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:22:22.922780 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:22.912482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41803 (local address 127.19.65.130:58125)
0619 10:22:22.912638 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:22.912677 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:22.913119 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:22.913362 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:22.913365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:22.913908 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:22.914089 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:22.914094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.915323 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.915325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:22.916668 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:22.916672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:22.917276 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:22.917277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:22.917373 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:22.917894 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:22.917902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:22.919247 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:22.921339 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:22.921346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:22.921348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:22.921578 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:22.921808 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:22.921809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:22.921810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:22.921919 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:22.922172 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:22.922178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:22.922313 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 10:22:22.922412 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 10:22:22.922558 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:22:22.923416 30255 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41803
I20260619 10:22:22.923535 30255 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:22.923723 30255 heartbeater.cc:511] Master 127.19.65.190:41803 requested a full tablet report, sending...
I20260619 10:22:22.924094 29930 ts_manager.cc:194] Registered new tserver with Master: 7eabf0f07c7f4c43ae93b46c70594705 (127.19.65.130:37337)
I20260619 10:22:22.924561 29930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:58125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:22:22.941363 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:22:23.021508 30260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:22:23.021685 30260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:22:23.021706 30260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:22:23.022722 30260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:22:23.022754 30260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:22:23.022779 30260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:22:23.024199 30260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:22:23.024365 30260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:22:23.024520 30260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:22:23.029325 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:23.029335 30266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:22:23.029350 30265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:22:23.029610 30260 server_base.cc:1034] running on GCE node
I20260619 10:22:23.029927 30260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:22:23.030158 30260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:22:23.031327 30260 hybrid_clock.cc:648] HybridClock initialized: now 1781864543031312 us; error 26 us; skew 500 ppm
Jun 19 10:22:23 dist-test-slave-xd2q krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864543, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:22:23.033816 30260 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:22:23.034253 30260 webserver.cc:460] Webserver started at http://127.19.65.131:42283/ using document root <none> and password file <none>
I20260619 10:22:23.034453 30260 fs_manager.cc:359] Metadata directory not provided
I20260619 10:22:23.034495 30260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:22:23.034580 30260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:22:23.035612 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "0082e511035f4c92bc03b2029a8f43bb"
format_stamp: "Formatted at 2026-06-19 10:22:23 on dist-test-slave-xd2q"
server_key: "4b71bc0781dfdad185eed2e0edbd1766"
server_key_iv: "8a87f54d35237449f98f4c7eaaa18238"
server_key_version: "encryptionkey@0"
I20260619 10:22:23.035913 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "0082e511035f4c92bc03b2029a8f43bb"
format_stamp: "Formatted at 2026-06-19 10:22:23 on dist-test-slave-xd2q"
server_key: "4b71bc0781dfdad185eed2e0edbd1766"
server_key_iv: "8a87f54d35237449f98f4c7eaaa18238"
server_key_version: "encryptionkey@0"
I20260619 10:22:23.037135 30260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:22:23.037801 30275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:23.037976 30260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:22:23.038061 30260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "0082e511035f4c92bc03b2029a8f43bb"
format_stamp: "Formatted at 2026-06-19 10:22:23 on dist-test-slave-xd2q"
server_key: "4b71bc0781dfdad185eed2e0edbd1766"
server_key_iv: "8a87f54d35237449f98f4c7eaaa18238"
server_key_version: "encryptionkey@0"
I20260619 10:22:23.038148 30260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:22:23.058194 30260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:22:23.060242 30260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:22:23.060364 30260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:22:23.060552 30260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:22:23.060892 30260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:22:23.060945 30260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:23.060982 30260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:22:23.061013 30260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:22:23.067044 30260 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:33807
I20260619 10:22:23.067106 30388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:33807 every 8 connection(s)
I20260619 10:22:23.067487 30260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:22:23 dist-test-slave-xd2q krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864543, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:22:23.075906 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 30260
I20260619 10:22:23.076031 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:22:23.076270 19718 external_mini_cluster.cc:1428] Setting key 615b962dabf5f0fbafc4f8cac7973d4c
I20260619 10:22:23.076999 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.068255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:55125 (local address 127.19.65.190:41803)
0619 10:22:23.068338 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.068340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.068516 (+ 176us) server_negotiation.cc:407] Connection header received
0619 10:22:23.068988 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.068990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.069002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.069059 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:23.069935 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.070414 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.071451 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.071722 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.071783 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.073857 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:23.073858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:23.073859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:23.073866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:23.075451 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:23.075853 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:23.075854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:23.075855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:23.075926 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:23.076230 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:23.076244 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:23.076244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:23.076382 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:23.076451 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.076802 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.076882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 10:22:23.077184 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.068213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41803 (local address 127.19.65.131:55125)
0619 10:22:23.068396 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.068429 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.068902 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.069110 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.069113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.069664 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:22:23.069847 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.069851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.070512 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.070513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.071357 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.071360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.071798 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.071799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.071898 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.072338 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:22:23.072346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:22:23.073753 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:22:23.075523 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:23.075528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:23.075530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:23.075772 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:23.076014 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:22:23.076015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:22:23.076015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:22:23.076151 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:22:23.076459 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:22:23.076465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:22:23.076659 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.076769 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.076933 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260619 10:22:23.077565 30389 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41803
I20260619 10:22:23.077636 30389 heartbeater.cc:464] Registering TS with master...
I20260619 10:22:23.077819 30389 heartbeater.cc:511] Master 127.19.65.190:41803 requested a full tablet report, sending...
I20260619 10:22:23.078166 29930 ts_manager.cc:194] Registered new tserver with Master: 0082e511035f4c92bc03b2029a8f43bb (127.19.65.131:33807)
I20260619 10:22:23.078531 29930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:55125
I20260619 10:22:23.088088 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:22:23.095279 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.089373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.19.65.190:41803)
0619 10:22:23.089443 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.089446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.089532 (+ 86us) server_negotiation.cc:407] Connection header received
0619 10:22:23.089627 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.089628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.089638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.089693 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:22:23.090301 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.090765 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.091552 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.091812 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.091897 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.092984 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:22:23.092985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:22:23.092985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:22:23.092992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:22:23.094128 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:23.094460 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:23.094461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:23.094461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:23.094515 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:22:23.094749 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:22:23.094750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:22:23.094750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:22:23.094897 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:22:23.095002 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.095104 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.095180 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:22:23.096768 29930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:22:23.098325 29930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:22:23.104900 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.100716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33807 (local address 127.0.0.1:40734)
0619 10:22:23.100992 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.101002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.101077 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.101615 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.101616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.101618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:23.101648 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.101650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102886 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.102887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.103840 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.103842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104589 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.104674 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.104688 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.104757 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.104804 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":75,"threads_started":1}
I20260619 10:22:23.105010 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.101114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.19.65.129:35345)
0619 10:22:23.101311 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.101316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.101328 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:22:23.101373 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.101375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.101391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.101493 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:23.101750 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102438 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.103869 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104676 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104756 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.104819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.104888 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.104931 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260619 10:22:23.105170 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.100797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.19.65.131:33807)
0619 10:22:23.101228 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.101232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.101239 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:22:23.101267 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.101269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.101281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.101382 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:23.101709 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102781 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.103918 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104480 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104587 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.104649 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.105056 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.105103 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":71,"threads_started":1}
I20260619 10:22:23.105029 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.100849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:35345 (local address 127.0.0.1:44782)
0619 10:22:23.101075 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.101084 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.101123 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.101639 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.101640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.101641 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:23.101691 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.101693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102536 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.102537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.103778 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.103780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104762 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.104822 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.104826 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.104878 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.104950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260619 10:22:23.105288 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.100678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:37337 (local address 127.0.0.1:52742)
0619 10:22:23.100846 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.100888 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.100967 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.101492 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.101495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.101497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:23.101559 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.101562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102869 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.102869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.103973 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.103975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104828 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.104914 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.104919 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.104950 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.105203 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3,"thread_start_us":43,"threads_started":1}
I20260619 10:22:23.105752 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.100876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.19.65.130:37337)
0619 10:22:23.101349 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.101352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.101362 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:23.101395 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.101397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.101411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.101486 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:23.101624 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.102758 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.104047 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.104737 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.105357 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.105450 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.105476 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.105670 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":92,"threads_started":1}
I20260619 10:22:23.106019 30323 tablet_service.cc:1467] Processing CreateTablet for tablet 981fe64b83ac48659ca920b88ead75f5 (DEFAULT_TABLE table=test-table [id=7b940ee991024d53be585e2eb5d2cc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:23.106359 30323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981fe64b83ac48659ca920b88ead75f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:23.106340 30189 tablet_service.cc:1467] Processing CreateTablet for tablet 981fe64b83ac48659ca920b88ead75f5 (DEFAULT_TABLE table=test-table [id=7b940ee991024d53be585e2eb5d2cc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:23.106659 30189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981fe64b83ac48659ca920b88ead75f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:23.107909 30055 tablet_service.cc:1467] Processing CreateTablet for tablet 981fe64b83ac48659ca920b88ead75f5 (DEFAULT_TABLE table=test-table [id=7b940ee991024d53be585e2eb5d2cc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:22:23.108222 30055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981fe64b83ac48659ca920b88ead75f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:22:23.109345 30406 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Bootstrap starting.
I20260619 10:22:23.109699 30407 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Bootstrap starting.
I20260619 10:22:23.110033 30406 tablet_bootstrap.cc:654] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:23.110348 30407 tablet_bootstrap.cc:654] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:23.110595 30406 log.cc:824] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:23.110910 30407 log.cc:824] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:23.111361 30406 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: No bootstrap required, opened a new log
I20260619 10:22:23.111433 30406 ts_tablet_manager.cc:1397] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:22:23.111589 30410 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: Bootstrap starting.
I20260619 10:22:23.111756 30407 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: No bootstrap required, opened a new log
I20260619 10:22:23.111862 30407 ts_tablet_manager.cc:1397] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:22:23.112269 30410 tablet_bootstrap.cc:654] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: Neither blocks nor log segments found. Creating new log.
I20260619 10:22:23.112723 30410 log.cc:824] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: Log is configured to *not* fsync() on all Append() calls
I20260619 10:22:23.113391 30410 tablet_bootstrap.cc:492] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: No bootstrap required, opened a new log
I20260619 10:22:23.113469 30410 ts_tablet_manager.cc:1397] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:22:23.113420 30406 raft_consensus.cc:348] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.113595 30406 raft_consensus.cc:374] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:23.113667 30406 raft_consensus.cc:729] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eabf0f07c7f4c43ae93b46c70594705, State: Initialized, Role: FOLLOWER
I20260619 10:22:23.113790 30406 consensus_queue.cc:260] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.113811 30407 raft_consensus.cc:348] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.114001 30407 raft_consensus.cc:374] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:23.114053 30407 raft_consensus.cc:729] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0082e511035f4c92bc03b2029a8f43bb, State: Initialized, Role: FOLLOWER
I20260619 10:22:23.114094 30406 ts_tablet_manager.cc:1428] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 10:22:23.114159 30407 consensus_queue.cc:260] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.114216 30255 heartbeater.cc:503] Master 127.19.65.190:41803 was elected leader, sending a full tablet report...
I20260619 10:22:23.114439 30407 ts_tablet_manager.cc:1428] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 10:22:23.114506 30389 heartbeater.cc:503] Master 127.19.65.190:41803 was elected leader, sending a full tablet report...
I20260619 10:22:23.116029 30410 raft_consensus.cc:348] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.116148 30410 raft_consensus.cc:374] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:22:23.116187 30410 raft_consensus.cc:729] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b68bc34826470aa945e9bedda85d02, State: Initialized, Role: FOLLOWER
I20260619 10:22:23.116290 30410 consensus_queue.cc:260] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.116504 30410 ts_tablet_manager.cc:1428] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:22:23.116597 30121 heartbeater.cc:503] Master 127.19.65.190:41803 was elected leader, sending a full tablet report...
W20260619 10:22:23.162395 30256 tablet.cc:2369] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:22:23.228413 30412 raft_consensus.cc:482] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:22:23.228525 30412 raft_consensus.cc:504] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.228968 30412 leader_election.cc:288] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b68bc34826470aa945e9bedda85d02 (127.19.65.129:35345), 0082e511035f4c92bc03b2029a8f43bb (127.19.65.131:33807)
I20260619 10:22:23.232015 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.229277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:39775 (local address 127.19.65.129:35345)
0619 10:22:23.229332 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.229334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.229344 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:22:23.229373 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.229374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.229382 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.229465 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:23.229625 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.230085 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231126 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.231659 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231747 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.231756 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.231913 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.231946 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:22:23.232028 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.229140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:35345 (local address 127.19.65.130:39775)
0619 10:22:23.229190 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.229203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.229253 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.229469 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.229470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.229472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:23.229541 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.229542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.230175 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.230175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.230999 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.231001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.231746 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.231837 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.231840 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.231908 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.231947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":60}
I20260619 10:22:23.232129 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.229154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:33807 (local address 127.19.65.130:36757)
0619 10:22:23.229345 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.229356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:22:23.229408 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:22:23.229570 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:22:23.229571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:22:23.229572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:22:23.229652 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.229653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.230304 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.230305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.231075 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:22:23.231077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.231842 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:22:23.231926 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.231932 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:22:23.232000 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:22:23.232043 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260619 10:22:23.232123 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.229344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:36757 (local address 127.19.65.131:33807)
0619 10:22:23.229425 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.229427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.229441 (+ 14us) server_negotiation.cc:407] Connection header received
0619 10:22:23.229472 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.229473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.229483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.229565 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:22:23.229729 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.230210 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231194 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.231732 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.231846 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.231855 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.232022 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.232061 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 10:22:23.232363 30073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981fe64b83ac48659ca920b88ead75f5" candidate_uuid: "7eabf0f07c7f4c43ae93b46c70594705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b68bc34826470aa945e9bedda85d02" is_pre_election: true
I20260619 10:22:23.232543 30073 raft_consensus.cc:2454] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eabf0f07c7f4c43ae93b46c70594705 in term 0.
I20260619 10:22:23.232591 30343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981fe64b83ac48659ca920b88ead75f5" candidate_uuid: "7eabf0f07c7f4c43ae93b46c70594705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0082e511035f4c92bc03b2029a8f43bb" is_pre_election: true
I20260619 10:22:23.232755 30343 raft_consensus.cc:2454] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eabf0f07c7f4c43ae93b46c70594705 in term 0.
I20260619 10:22:23.232789 30143 leader_election.cc:302] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b68bc34826470aa945e9bedda85d02, 7eabf0f07c7f4c43ae93b46c70594705; no voters:
I20260619 10:22:23.232929 30412 raft_consensus.cc:2781] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:22:23.233001 30412 raft_consensus.cc:482] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:22:23.233019 30412 raft_consensus.cc:3037] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:23.233665 30412 raft_consensus.cc:504] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.233827 30412 leader_election.cc:288] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [CANDIDATE]: Term 1 election: Requested vote from peers 51b68bc34826470aa945e9bedda85d02 (127.19.65.129:35345), 0082e511035f4c92bc03b2029a8f43bb (127.19.65.131:33807)
I20260619 10:22:23.234050 30073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981fe64b83ac48659ca920b88ead75f5" candidate_uuid: "7eabf0f07c7f4c43ae93b46c70594705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b68bc34826470aa945e9bedda85d02"
I20260619 10:22:23.234158 30073 raft_consensus.cc:3037] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:23.234170 30343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981fe64b83ac48659ca920b88ead75f5" candidate_uuid: "7eabf0f07c7f4c43ae93b46c70594705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0082e511035f4c92bc03b2029a8f43bb"
I20260619 10:22:23.234238 30343 raft_consensus.cc:3037] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:22:23.235158 30073 raft_consensus.cc:2454] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eabf0f07c7f4c43ae93b46c70594705 in term 1.
I20260619 10:22:23.235176 30343 raft_consensus.cc:2454] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eabf0f07c7f4c43ae93b46c70594705 in term 1.
I20260619 10:22:23.235468 30142 leader_election.cc:302] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0082e511035f4c92bc03b2029a8f43bb, 7eabf0f07c7f4c43ae93b46c70594705; no voters:
I20260619 10:22:23.235597 30412 raft_consensus.cc:2781] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:22:23.235757 30412 raft_consensus.cc:686] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [term 1 LEADER]: Becoming Leader. State: Replica: 7eabf0f07c7f4c43ae93b46c70594705, State: Running, Role: LEADER
I20260619 10:22:23.235854 30412 consensus_queue.cc:237] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } }
I20260619 10:22:23.236689 29928 catalog_manager.cc:5495] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eabf0f07c7f4c43ae93b46c70594705 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "7eabf0f07c7f4c43ae93b46c70594705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eabf0f07c7f4c43ae93b46c70594705" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260619 10:22:23.251659 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.249250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.19.65.130:37337)
0619 10:22:23.249293 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:22:23.249296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:22:23.249304 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:22:23.249338 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:22:23.249338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:22:23.249352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:22:23.249426 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:22:23.249577 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.250025 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.250809 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:22:23.250991 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:22:23.251062 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:22:23.251223 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:22:23.251382 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:22:23.251474 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:22:23.251493 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 10:22:23.251534 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:22:23.253460 30073 raft_consensus.cc:1261] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02 [term 1 FOLLOWER]: Refusing update from remote peer 7eabf0f07c7f4c43ae93b46c70594705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:23.253479 30343 raft_consensus.cc:1261] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 1 FOLLOWER]: Refusing update from remote peer 7eabf0f07c7f4c43ae93b46c70594705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:22:23.253808 30412 consensus_queue.cc:1035] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b68bc34826470aa945e9bedda85d02" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:23.253935 30416 consensus_queue.cc:1035] T 981fe64b83ac48659ca920b88ead75f5 P 7eabf0f07c7f4c43ae93b46c70594705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0082e511035f4c92bc03b2029a8f43bb" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:22:23.256524 30420 mvcc.cc:205] Tried to move back new op lower bound from 7298517169163526144 to 7298517169094660096. Current Snapshot: MvccSnapshot[applied={T|T < 7298517169163526144}]
I20260619 10:22:23.257118 30421 mvcc.cc:205] Tried to move back new op lower bound from 7298517169163526144 to 7298517169094660096. Current Snapshot: MvccSnapshot[applied={T|T < 7298517169163526144}]
I20260619 10:22:23.258395 29928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:23.258507 29928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:22:23.259284 29928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a5732244b8349568613b109c107c1e7: Sending DeleteTablet for 3 replicas of tablet 981fe64b83ac48659ca920b88ead75f5
I20260619 10:22:23.259630 30189 tablet_service.cc:1514] Processing DeleteTablet for tablet 981fe64b83ac48659ca920b88ead75f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:23 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52742
I20260619 10:22:23.259662 30323 tablet_service.cc:1514] Processing DeleteTablet for tablet 981fe64b83ac48659ca920b88ead75f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:23 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:40734
I20260619 10:22:23.259719 30055 tablet_service.cc:1514] Processing DeleteTablet for tablet 981fe64b83ac48659ca920b88ead75f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:22:23 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:44782
I20260619 10:22:23.259917 30431 tablet_replica.cc:330] T 981fe64b83ac48659ca920b88ead75f5 P 51b68bc34826470aa945e9bedda85d02: stopping tablet replica
I20260619 10:22:23.259980 30430 tablet_replica.cc:330] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: stopping tablet replica
I20260619 10:22:23.260087 30430 raft_consensus.cc:2229] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:22:23.260192 30430 raft_consensus.cc:2258] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:22:23.260305 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29992
I20260619 10:22:23.262596 30430 ts_tablet_manager.cc:1905] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:22:23.264748 30430 ts_tablet_manager.cc:1918] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:22:23.264833 30430 log.cc:1196] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/981fe64b83ac48659ca920b88ead75f5
I20260619 10:22:23.265165 30430 ts_tablet_manager.cc:1939] T 981fe64b83ac48659ca920b88ead75f5 P 0082e511035f4c92bc03b2029a8f43bb: Deleting consensus metadata
I20260619 10:22:23.265607 29914 catalog_manager.cc:4841] TS 0082e511035f4c92bc03b2029a8f43bb (127.19.65.131:33807): tablet 981fe64b83ac48659ca920b88ead75f5 (table test-table [id=7b940ee991024d53be585e2eb5d2cc60]) successfully deleted
W20260619 10:22:23.266100 29915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:35345 (error 108)
I20260619 10:22:23.266649 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.266502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:35345 (local address 127.0.0.1:44790)
0619 10:22:23.266567 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.266609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 10:22:23.266710 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 30126
W20260619 10:22:23.268101 29915 catalog_manager.cc:4584] TS 51b68bc34826470aa945e9bedda85d02 (127.19.65.129:35345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 981fe64b83ac48659ca920b88ead75f5: Network error: Client connection negotiation failed: client connection to 127.19.65.129:35345: connect: Connection refused (error 111)
I20260619 10:22:23.271270 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:22:23.271163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:37337 (local address 127.0.0.1:52758)
0619 10:22:23.271215 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:22:23.271240 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.130:37337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:22:23.271692 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 30260
I20260619 10:22:23.276078 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 29898
2026-06-19T10:22:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1953 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:22:23 dist-test-slave-xd2q krb5kdc[30435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:22:23 dist-test-slave-xd2q krb5kdc[30435](info): set up 2 sockets
Jun 19 10:22:23 dist-test-slave-xd2q krb5kdc[30435](info): commencing operation
krb5kdc: starting...
W20260619 10:22:24.324115 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:22:24 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:22:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
I20260619 10:22:24.486114 19718 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskadFLwT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/postgres -l logfile start
2026-06-19 10:22:26.768 UTC [30465] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 10:22:26.768 UTC [30465] LOG: listening on IPv4 address "127.19.65.148", port 47539
2026-06-19 10:22:26.771 UTC [30465] LOG: listening on Unix socket "/tmp/.s.PGSQL.47539"
2026-06-19 10:22:26.807 UTC [30468] LOG: database system was shut down at 2026-06-19 10:22:25 UTC
2026-06-19 10:22:26.811 UTC [30465] LOG: database system is ready to accept connections
I20260619 10:22:27.748158 19718 mini_postgres.cc:96] Postgres bound to 47539
2026-06-19 10:22:27.754 UTC [30476] FATAL: database "slave" does not exist
127.19.65.148:47539 - accepting connections
I20260619 10:22:27.754616 19718 mini_ranger.cc:161] Starting Ranger...
I20260619 10:22:27.768615 19718 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 10:22:28.195571 19718 mini_ranger.cc:87] Created ranger Postgres database
I20260619 10:22:28.195636 19718 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin
2026-06-19 10:22:28,462 [I] DB FLAVOR :POSTGRES
2026-06-19 10:22:28,462 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:22:28,462 [I] Checking connection..
2026-06-19 10:22:28,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 10:22:28,713 [I] Checking connection passed.
2026-06-19 10:22:28,713 [I] --------- Verifying version history table ---------
2026-06-19 10:22:28,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:22:28,968 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:22:28,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:22:29,216 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:22:29,216 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 10:22:29,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 10:22:29.463 UTC [30540] WARNING: there is no transaction in progress
2026-06-19 10:22:29,469 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 10:22:29,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:22:29,723 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 10:22:29,723 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:22:29,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 10:22:29,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 10:22:30,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xd2q.c.gcp-upstream.internal','N') ;"
2026-06-19 10:22:30,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:22:30,667 [I] Table x_portal_user does not exist in database ranger
2026-06-19 10:22:30,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:22:30,896 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 10:22:30,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:22:31,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:22:31,376 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 10:22:31,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 10:22:32.251 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.266 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.281 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.297 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.312 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.329 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.379 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.386 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.398 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.407 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.417 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.430 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.439 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.445 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.451 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:32.457 UTC [30648] WARNING: there is no transaction in progress
2026-06-19 10:22:33,052 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 10:22:33,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 10:22:33,287 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 10:22:33,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:22:33,530 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 10:22:33,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:22:33,782 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 10:22:33,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:22:34,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:22:34,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xd2q.c.gcp-upstream.internal';"
2026-06-19 10:22:34,460 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 10:22:34,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 10:22:34,705 [I] DB_PATCHES have already been applied
I20260619 10:22:34.711197 19718 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskadFLwT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 10:22:34.711254 19718 mini_ranger.cc:193] Using host: 127.19.65.148
I20260619 10:22:34.714071 19718 mini_ranger.cc:227] Ranger admin URL: http://127.19.65.148:41793
Jun 19, 2026 10:22:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:22:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 10:22:35.817633 19718 mini_ranger.cc:160] Time spent starting Ranger: real 8.063s user 0.004s sys 0.004s
Jun 19, 2026 10:22:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:22:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 10:22:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:22:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:22:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 10:22:55.858786 19718 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.65.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 10:22:55 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864575, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 10:22:55.935657 19718 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 10:22:55.944741 19718 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 10:22:56.761191 19718 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 10:22:56.761270 19718 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms
I20260619 10:22:56.761292 19718 mini_ranger_kms.cc:227] Using postgres at 127.19.65.148:47539
2026-06-19 10:22:56,942 [I] DB FLAVOR :POSTGRES
2026-06-19 10:22:56,942 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:22:56,943 [I] Checking connection
2026-06-19 10:22:56,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 10:22:57,219 [I] connection success
2026-06-19 10:22:57,220 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 10:22:57,220 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 10:22:57,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 10:22:57,481 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 10:22:57,481 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:22:57,482 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 10:22:57,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:47539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 10:22:57,720 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 10:22:58.147136 19718 mini_ranger_kms.cc:310] Created kms service
I20260619 10:22:58.318113 19718 mini_ranger_kms.cc:326] Created kudu user
I20260619 10:22:58.358565 19718 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 10:22:58 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864575, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:22:58.733558 19718 mini_ranger_kms.cc:384] Added ranger policy
I20260619 10:22:58.733834 19718 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms:/tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 10:22:58.733887 19718 mini_ranger_kms.cc:240] Using host: 127.19.65.148
I20260619 10:22:58.737416 19718 mini_ranger_kms.cc:276] Ranger KMS PID: 30828
I20260619 10:22:58.737528 19718 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.65.148:51269
Jun 19, 2026 10:22:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:22:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:22:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:22:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:23:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:23:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 10:23:00 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:01.001070 19718 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.065s user 0.000s sys 0.152s
I20260619 10:23:01.001201 19718 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 10:23:01.001299 19718 mini_ranger_kms.cc:401] 127.19.65.148:51269/kms/v1/keys
Jun 19 10:23:01 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Jun 19 10:23:01 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Jun 19 10:23:04 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:23:04.068059 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33477
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:33477
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:51269/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:04.148041 30879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:04.148263 30879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:04.148299 30879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:04.149444 30879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:04.149525 30879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:04.149547 30879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:04.149567 30879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:04.149595 30879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:04.151091 30879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.65.148:51269/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:33477
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:04.151382 30879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:04.151618 30879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:04.157382 30885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:04.157475 30879 server_base.cc:1034] running on GCE node
W20260619 10:23:04.157619 30884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:04.157413 30887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:04.158819 30879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:04.159101 30879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:04.160260 30879 hybrid_clock.cc:648] HybridClock initialized: now 1781864584160243 us; error 38 us; skew 500 ppm
Jun 19 10:23:04 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864584, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:04.162803 30879 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:04.163328 30879 webserver.cc:460] Webserver started at http://127.19.65.190:36913/ using document root <none> and password file <none>
I20260619 10:23:04.163648 30879 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:04.163703 30879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:04.163821 30879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:04 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864584, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:04.214773 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d"
format_stamp: "Formatted at 2026-06-19 10:23:04 on dist-test-slave-xd2q"
server_key: "36a13d8632f74870f35f3afbd9228df7"
server_key_iv: "94c3349cb6aee9ddad0f04502f9fb8db"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:04.215178 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d"
format_stamp: "Formatted at 2026-06-19 10:23:04 on dist-test-slave-xd2q"
server_key: "36a13d8632f74870f35f3afbd9228df7"
server_key_iv: "94c3349cb6aee9ddad0f04502f9fb8db"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:04.216650 30879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:04.227488 30894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:04.227679 30879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:04.227767 30879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d"
format_stamp: "Formatted at 2026-06-19 10:23:04 on dist-test-slave-xd2q"
server_key: "36a13d8632f74870f35f3afbd9228df7"
server_key_iv: "94c3349cb6aee9ddad0f04502f9fb8db"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:04.227844 30879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:04.238443 30879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:04.240727 30879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:04.240916 30879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:04.245569 30879 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:33477
I20260619 10:23:04.245592 30946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:33477 every 8 connection(s)
I20260619 10:23:04.245944 30879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:04.246594 30947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:04.249270 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: Bootstrap starting.
I20260619 10:23:04.249801 30947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:04.250077 30947 log.cc:824] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:04.250706 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: No bootstrap required, opened a new log
I20260619 10:23:04.252763 30947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } }
I20260619 10:23:04.252933 30947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:04.252995 30947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594fab5b6c3a4bcbbee1bf48526d1f7d, State: Initialized, Role: FOLLOWER
I20260619 10:23:04.253082 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 30879
I20260619 10:23:04.253194 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:23:04.253221 30947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } }
I20260619 10:23:04.253306 30947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:04.253340 30947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:04.253360 19718 external_mini_cluster.cc:1428] Setting key 1c8b17ac18dd625ad97510d1f308a7dd
I20260619 10:23:04.253387 30947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:04.254482 30947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } }
I20260619 10:23:04.254582 30947 leader_election.cc:302] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594fab5b6c3a4bcbbee1bf48526d1f7d; no voters:
I20260619 10:23:04.254732 30947 leader_election.cc:288] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:23:04.254884 30952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:04.255023 30947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:04.255048 30952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [term 1 LEADER]: Becoming Leader. State: Replica: 594fab5b6c3a4bcbbee1bf48526d1f7d, State: Running, Role: LEADER
I20260619 10:23:04.255159 30952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } }
I20260619 10:23:04.255453 30952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594fab5b6c3a4bcbbee1bf48526d1f7d. Latest consensus state: current_term: 1 leader_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } } }
I20260619 10:23:04.255534 30952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:04.255726 30956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:04.256234 30956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:04.257452 30947 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 10:23:04.255463 30954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fab5b6c3a4bcbbee1bf48526d1f7d" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33477 } } }
I20260619 10:23:04.257563 30954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:04.258494 30963 mvcc.cc:205] Tried to move back new op lower bound from 7298517337116876800 to 7298517337109385216. Current Snapshot: MvccSnapshot[applied={T|T < 7298517337116876800}]
I20260619 10:23:04.263281 30956 catalog_manager.cc:1347] Generated new cluster ID: 234349e64c294014a3500bd7f6598a65
I20260619 10:23:04.263356 30956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:23:04 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:04.272423 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:04.253867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.19.65.190:33477)
0619 10:23:04.254135 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:04.254146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:04.254420 (+ 274us) server_negotiation.cc:407] Connection header received
0619 10:23:04.254541 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:04.254549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:04.254605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:04.254762 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:04.255547 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:04.257391 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:04.258452 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:04.261331 (+ 2879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:04.261499 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:04.267955 (+ 6456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:04.267960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:04.267960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:04.267972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:04.270715 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:04.271145 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:04.271148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:04.271148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:04.271253 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:04.271503 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:04.271503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:04.271504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:04.271819 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:04.271997 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:04.272096 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:23:04.272218 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260619 10:23:04.284068 30956 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:23:04.284853 30956 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:04.288522 30956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: Generated new TSK 0
I20260619 10:23:04.288827 30956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:23:05.028475 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:51269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:05.112723 30992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:05.112908 30992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:05.112931 30992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:05.114053 30992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:05.114092 30992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:05.114117 30992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:23:05.115619 30992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.65.148:51269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:05.115832 30992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:05.116050 30992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:05.121814 30998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:05.122104 31000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:05.122241 30992 server_base.cc:1034] running on GCE node
W20260619 10:23:05.122745 30997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:05.122990 30992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:05.123222 30992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:05.130599 30992 hybrid_clock.cc:648] HybridClock initialized: now 1781864585130552 us; error 56 us; skew 500 ppm
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:05.133188 30992 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:23:05.133634 30992 webserver.cc:460] Webserver started at http://127.19.65.129:43693/ using document root <none> and password file <none>
I20260619 10:23:05.133845 30992 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:05.133908 30992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:05.134045 30992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:05.148537 30992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "a47ec762b9484d1ab33a9660deb8bb8e"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "8e2bd8c2b03e184740fabbdd9d3f7176"
server_key_iv: "edbc077b682be8b7cb65b1faadf11b45"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.148962 30992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "a47ec762b9484d1ab33a9660deb8bb8e"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "8e2bd8c2b03e184740fabbdd9d3f7176"
server_key_iv: "edbc077b682be8b7cb65b1faadf11b45"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.150684 30992 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 10:23:05.165274 31007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.165488 30992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:23:05.165567 30992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "a47ec762b9484d1ab33a9660deb8bb8e"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "8e2bd8c2b03e184740fabbdd9d3f7176"
server_key_iv: "edbc077b682be8b7cb65b1faadf11b45"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.165658 30992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:05.179993 30992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:05.182384 30992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:05.182574 30992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:05.182828 30992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:05.183241 30992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:05.183339 30992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.183399 30992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:05.183439 30992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.191040 30992 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42401
I20260619 10:23:05.191195 31122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:42401 every 8 connection(s)
I20260619 10:23:05.191514 30992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:23:05.197697 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 30992
I20260619 10:23:05.197795 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:23:05.197981 19718 external_mini_cluster.cc:1428] Setting key a401f2e89a14326d6ad091f7b7155b5c
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:05.202042 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.192334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:53541 (local address 127.19.65.190:33477)
0619 10:23:05.193160 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.193162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.193168 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:23:05.193190 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.193190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.193199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.193256 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:05.194500 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.195061 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.196439 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.196750 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.196814 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.199124 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:05.199126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:05.199126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:05.199141 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:05.200665 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.201063 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.201064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.201064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.201131 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.201384 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.201385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.201385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.201503 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:05.201578 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.201848 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.201928 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":100,"threads_started":1}
I20260619 10:23:05.202239 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.192308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33477 (local address 127.19.65.129:53541)
0619 10:23:05.192497 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.192543 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.193001 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.193412 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.193417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.194047 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:05.194399 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.194407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.195156 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.195157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.196335 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.196337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.196890 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.196891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.197004 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.197580 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:05.197590 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:05.199025 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:05.200765 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.200772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.200774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.200987 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.201205 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.201206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.201206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.201317 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.201570 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:05.201576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:05.201719 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.201828 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.201971 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260619 10:23:05.202777 31123 heartbeater.cc:347] Connected to a master server at 127.19.65.190:33477
I20260619 10:23:05.202900 31123 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:05.203083 31123 heartbeater.cc:511] Master 127.19.65.190:33477 requested a full tablet report, sending...
I20260619 10:23:05.203685 30911 ts_manager.cc:194] Registered new tserver with Master: a47ec762b9484d1ab33a9660deb8bb8e (127.19.65.129:42401)
I20260619 10:23:05.204375 30911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:53541
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:23:05.228298 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:51269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:05.335888 31129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:05.336145 31129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:05.336177 31129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:05.337796 31129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:05.337847 31129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:05.337882 31129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:23:05.340178 31129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.65.148:51269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:05.340406 31129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:05.340675 31129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:05.347064 31139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:05.347532 31129 server_base.cc:1034] running on GCE node
W20260619 10:23:05.348018 31136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:05.348155 31137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:05.348418 31129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:05.348690 31129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:05.349874 31129 hybrid_clock.cc:648] HybridClock initialized: now 1781864585349860 us; error 29 us; skew 500 ppm
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:05.352986 31129 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:23:05.353510 31129 webserver.cc:460] Webserver started at http://127.19.65.130:46525/ using document root <none> and password file <none>
I20260619 10:23:05.353752 31129 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:05.353871 31129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:05.354048 31129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:05.372514 31129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "e3a624c6189049a1856d4ec026c966cc"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "177e818477ae2f14e3e6ab7c3711f638"
server_key_iv: "4ae12334188c0301329663f5e5897ef9"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.372967 31129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "e3a624c6189049a1856d4ec026c966cc"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "177e818477ae2f14e3e6ab7c3711f638"
server_key_iv: "4ae12334188c0301329663f5e5897ef9"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.376827 31129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260619 10:23:05.392624 31146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.392871 31129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:05.392966 31129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "e3a624c6189049a1856d4ec026c966cc"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "177e818477ae2f14e3e6ab7c3711f638"
server_key_iv: "4ae12334188c0301329663f5e5897ef9"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.393051 31129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:05.413312 31129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:05.415498 31129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:05.415639 31129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:05.415854 31129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:05.416198 31129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:05.416265 31129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.416303 31129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:05.416332 31129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.424111 31129 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:43101
I20260619 10:23:05.424193 31259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:43101 every 8 connection(s)
I20260619 10:23:05.424561 31129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:23:05.425202 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 31129
I20260619 10:23:05.425310 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:23:05.425515 19718 external_mini_cluster.cc:1428] Setting key 3d54abae5d84053ec9cc81561d3bdc12
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:05.435940 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.425652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:55677 (local address 127.19.65.190:33477)
0619 10:23:05.425699 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.425701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.425713 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:23:05.426331 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.426332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.426347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.426421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:05.427120 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.427821 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.429257 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.429547 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.429628 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.432378 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:05.432379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:05.432380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:05.432388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:05.434193 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.434719 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.434720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.434721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.434788 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.435076 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.435077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.435077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.435213 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:05.435261 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.435710 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.435810 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:23:05.436080 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.425352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33477 (local address 127.19.65.130:55677)
0619 10:23:05.425550 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.425600 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.426216 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.426509 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.426513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.426757 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:05.427007 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.427013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.427915 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.427916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.429154 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.429157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.429649 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.429651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.429787 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.430327 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:05.430336 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:05.432275 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:05.434270 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.434276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.434278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.434602 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.434868 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.434869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.434869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.434994 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.435352 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:05.435358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:05.435513 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.435662 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.435831 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260619 10:23:05.437141 31260 heartbeater.cc:347] Connected to a master server at 127.19.65.190:33477
I20260619 10:23:05.437243 31260 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:23:05.437490 31260 heartbeater.cc:511] Master 127.19.65.190:33477 requested a full tablet report, sending...
I20260619 10:23:05.437915 30911 ts_manager.cc:194] Registered new tserver with Master: e3a624c6189049a1856d4ec026c966cc (127.19.65.130:43101)
I20260619 10:23:05.438426 30911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:55677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:23:05.462954 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:51269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:05.584232 31266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:05.584501 31266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:05.584539 31266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:05.586167 31266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:05.586225 31266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:05.586267 31266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:23:05.588397 31266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.65.148:51269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:33477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:05.588636 31266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:05.588936 31266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:05.595238 31272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:05.595548 31274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:05.595263 31271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:05.596472 31266 server_base.cc:1034] running on GCE node
I20260619 10:23:05.596709 31266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:05.597034 31266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:05.598212 31266 hybrid_clock.cc:648] HybridClock initialized: now 1781864585598201 us; error 29 us; skew 500 ppm
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:05.600841 31266 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:23:05.601317 31266 webserver.cc:460] Webserver started at http://127.19.65.131:45877/ using document root <none> and password file <none>
I20260619 10:23:05.601553 31266 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:05.601631 31266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:05.601771 31266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:05.617666 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "cd7c085aea234bf487195f81e99ca6c1"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "6d2288062e5581cb864cb09a2c0d0992"
server_key_iv: "29a5e837b3ddb1a89162a68026621853"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.618180 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "cd7c085aea234bf487195f81e99ca6c1"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "6d2288062e5581cb864cb09a2c0d0992"
server_key_iv: "29a5e837b3ddb1a89162a68026621853"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.622318 31266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 10:23:05.635090 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.635612 31266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:05.635728 31266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "cd7c085aea234bf487195f81e99ca6c1"
format_stamp: "Formatted at 2026-06-19 10:23:05 on dist-test-slave-xd2q"
server_key: "6d2288062e5581cb864cb09a2c0d0992"
server_key_iv: "29a5e837b3ddb1a89162a68026621853"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:05.635841 31266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:05.653729 31266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:05.656818 31266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:05.657019 31266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:05.657337 31266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:05.657801 31266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:05.657861 31266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.657927 31266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:05.657956 31266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:05.664439 31266 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:42775
I20260619 10:23:05.664891 31266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:23:05.669178 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 31266
I20260619 10:23:05.669263 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:23:05.669432 19718 external_mini_cluster.cc:1428] Setting key 4708a22c047fabe1ac669ab0062723b8
I20260619 10:23:05.671016 31394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:42775 every 8 connection(s)
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:05.681403 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.670994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:54823 (local address 127.19.65.190:33477)
0619 10:23:05.671040 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.671042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.671225 (+ 183us) server_negotiation.cc:407] Connection header received
0619 10:23:05.671874 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.671875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.671886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.671914 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:05.672729 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.673462 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.674973 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.675310 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.675355 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.677889 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:05.677891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:05.677891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:05.677899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:05.679655 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.680131 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.680132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.680132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.680198 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.680507 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.680509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.680509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.680638 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:05.680733 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.681231 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.681308 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:23:05.681893 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.670749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33477 (local address 127.19.65.131:54823)
0619 10:23:05.671086 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.671148 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.671757 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.672124 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.672128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.672354 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:05.672611 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.672617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.673575 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.673576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.674870 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.674872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.675508 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.675509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.675654 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.676229 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:05.676239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:05.677789 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:05.679748 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.679754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.679757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.680044 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.680274 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:05.680275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:05.680275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:05.680423 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:05.680831 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:05.680837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:05.680988 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.681526 (+ 538us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.681683 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":73,"threads_started":1}
I20260619 10:23:05.682329 31395 heartbeater.cc:347] Connected to a master server at 127.19.65.190:33477
I20260619 10:23:05.682462 31395 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:05.682680 31395 heartbeater.cc:511] Master 127.19.65.190:33477 requested a full tablet report, sending...
I20260619 10:23:05.683123 30911 ts_manager.cc:194] Registered new tserver with Master: cd7c085aea234bf487195f81e99ca6c1 (127.19.65.131:42775)
I20260619 10:23:05.683633 30911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:54823
I20260619 10:23:05.691949 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:23:05.701565 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.693393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.19.65.190:33477)
0619 10:23:05.693437 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.693439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.693762 (+ 323us) server_negotiation.cc:407] Connection header received
0619 10:23:05.693819 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.693820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.693830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.693893 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:05.694690 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.695474 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.696801 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.697034 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.697146 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.698723 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:05.698724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:05.698725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:05.698732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:05.700284 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.700740 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.700741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.700742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.700815 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:05.701035 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:05.701036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:05.701037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:05.701147 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:05.701251 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.701381 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.701456 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:23:05.703348 30911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:23:05.705606 30911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:23:05.714270 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.709476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42775 (local address 127.0.0.1:33494)
0619 10:23:05.709726 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.709766 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.709829 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.710199 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.710202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.710205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:05.710494 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.710496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.711729 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.711730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.713086 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.713088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.714015 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.714016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.714098 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.714112 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.714155 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.714198 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260619 10:23:05.714926 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.709565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.19.65.131:42775)
0619 10:23:05.709974 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.709979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.709989 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:05.710026 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.710028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.710044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.710113 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:05.710549 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.711640 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.713212 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.713930 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.714699 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.714795 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.714819 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.714853 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":66,"threads_started":1}
I20260619 10:23:05.715746 31325 tablet_service.cc:1467] Processing CreateTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 (DEFAULT_TABLE table=test-table [id=82dd3bcf15a3410abefcb78ce3e32310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:05.716079 31325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c963acf30e2149e2bd7290e1ce9288a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:05.718596 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.709695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:43101 (local address 127.0.0.1:39678)
0619 10:23:05.710596 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.710604 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.710642 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.711053 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.711054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.711056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:05.711099 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.711101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.715618 (+ 4517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.715619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.716624 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.716627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.718383 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.718384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.718445 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.718448 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.718498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.718532 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260619 10:23:05.719664 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.709868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42401 (local address 127.0.0.1:35460)
0619 10:23:05.710708 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.710717 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.710750 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.712506 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.712506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.712508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:05.712540 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.712542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.713724 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.713725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.715081 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.715083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.719486 (+ 4403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.719487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.719545 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.719548 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.719590 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.719618 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":73,"threads_started":1}
I20260619 10:23:05.719987 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.709807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.19.65.130:43101)
0619 10:23:05.710860 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.710864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.710873 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:05.710906 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.710908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.710922 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.710987 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:05.711382 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.715530 (+ 4148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.716741 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.718193 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.718276 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.718348 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.719848 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.719904 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":71,"threads_started":1}
I20260619 10:23:05.720392 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.710062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.19.65.129:42401)
0619 10:23:05.711196 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.711199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.711207 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:23:05.711235 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.711237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.711250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.712449 (+ 1199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:05.712609 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.713639 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.717513 (+ 3874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.719162 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.719226 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.720258 (+ 1032us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.720291 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.720331 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":58,"threads_started":1}
I20260619 10:23:05.720577 31194 tablet_service.cc:1467] Processing CreateTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 (DEFAULT_TABLE table=test-table [id=82dd3bcf15a3410abefcb78ce3e32310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:05.720916 31194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c963acf30e2149e2bd7290e1ce9288a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:05.721437 31057 tablet_service.cc:1467] Processing CreateTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 (DEFAULT_TABLE table=test-table [id=82dd3bcf15a3410abefcb78ce3e32310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:05.721714 31057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c963acf30e2149e2bd7290e1ce9288a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:05.724220 31414 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Bootstrap starting.
I20260619 10:23:05.725279 31414 tablet_bootstrap.cc:654] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:05.725999 31415 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Bootstrap starting.
I20260619 10:23:05.726039 31414 log.cc:824] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:05.726625 31415 tablet_bootstrap.cc:654] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:05.726915 31415 log.cc:824] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:05.727257 31413 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Bootstrap starting.
I20260619 10:23:05.727641 31415 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: No bootstrap required, opened a new log
I20260619 10:23:05.727725 31415 ts_tablet_manager.cc:1397] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:23:05.729548 31415 raft_consensus.cc:348] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.729699 31415 raft_consensus.cc:374] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:05.730754 31413 tablet_bootstrap.cc:654] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:05.731063 31413 log.cc:824] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:05.731343 31415 raft_consensus.cc:729] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ec762b9484d1ab33a9660deb8bb8e, State: Initialized, Role: FOLLOWER
I20260619 10:23:05.731555 31415 consensus_queue.cc:260] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.731835 31415 ts_tablet_manager.cc:1428] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 10:23:05.732208 31413 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: No bootstrap required, opened a new log
I20260619 10:23:05.732296 31413 ts_tablet_manager.cc:1397] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 10:23:05.733592 31413 raft_consensus.cc:348] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.733733 31413 raft_consensus.cc:374] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:05.733836 31123 heartbeater.cc:503] Master 127.19.65.190:33477 was elected leader, sending a full tablet report...
I20260619 10:23:05.733841 31413 raft_consensus.cc:729] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c085aea234bf487195f81e99ca6c1, State: Initialized, Role: FOLLOWER
I20260619 10:23:05.733978 31413 consensus_queue.cc:260] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.734530 31413 ts_tablet_manager.cc:1428] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:23:05.734681 31395 heartbeater.cc:503] Master 127.19.65.190:33477 was elected leader, sending a full tablet report...
I20260619 10:23:05.735178 31414 tablet_bootstrap.cc:492] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: No bootstrap required, opened a new log
I20260619 10:23:05.735260 31414 ts_tablet_manager.cc:1397] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 10:23:05.737053 31414 raft_consensus.cc:348] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.737227 31414 raft_consensus.cc:374] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:05.737286 31414 raft_consensus.cc:729] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a624c6189049a1856d4ec026c966cc, State: Initialized, Role: FOLLOWER
I20260619 10:23:05.737429 31414 consensus_queue.cc:260] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.737679 31414 ts_tablet_manager.cc:1428] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:23:05.737730 31260 heartbeater.cc:503] Master 127.19.65.190:33477 was elected leader, sending a full tablet report...
Jun 19 10:23:05 dist-test-slave-xd2q krb5kdc[30435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864585, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
W20260619 10:23:05.920818 31396 tablet.cc:2369] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:23:05.925647 31261 tablet.cc:2369] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:05.933099 31421 raft_consensus.cc:482] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:23:05.933223 31421 raft_consensus.cc:504] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.933589 31421 leader_election.cc:288] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7c085aea234bf487195f81e99ca6c1 (127.19.65.131:42775), a47ec762b9484d1ab33a9660deb8bb8e (127.19.65.129:42401)
I20260619 10:23:05.937567 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.933782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:42775 (local address 127.19.65.130:35985)
0619 10:23:05.934109 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.934121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.934173 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.934364 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.934365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.934384 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:05.934454 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.934456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.935399 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.935399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.936561 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.936563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.937344 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.937345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.937414 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.937418 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.937463 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.937500 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260619 10:23:05.937819 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.934257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59975 (local address 127.19.65.129:42401)
0619 10:23:05.934306 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.934308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.934423 (+ 115us) server_negotiation.cc:407] Connection header received
0619 10:23:05.934511 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.934512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.934520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.934548 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:05.934795 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.935543 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.936860 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.937462 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.937504 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.937511 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.937719 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.937753 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:23:05.938047 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.933875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42401 (local address 127.19.65.130:59975)
0619 10:23:05.934358 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:05.934366 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:05.934444 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:05.934643 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:05.934643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:05.934645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:05.934724 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.934725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.935645 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.935645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.936763 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:05.936765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.937581 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.937582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:05.937647 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.937650 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:23:05.937923 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 10:23:05.937962 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":55,"threads_started":1}
I20260619 10:23:05.938133 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.934055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:35985 (local address 127.19.65.131:42775)
0619 10:23:05.934096 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.934098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.934160 (+ 62us) server_negotiation.cc:407] Connection header received
0619 10:23:05.934227 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.934228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.934237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.934296 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:05.934529 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.935247 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.936686 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.937203 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.937304 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.937313 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.938027 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.938067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:23:05.938426 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c963acf30e2149e2bd7290e1ce9288a8" candidate_uuid: "e3a624c6189049a1856d4ec026c966cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c085aea234bf487195f81e99ca6c1" is_pre_election: true
I20260619 10:23:05.938477 31077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c963acf30e2149e2bd7290e1ce9288a8" candidate_uuid: "e3a624c6189049a1856d4ec026c966cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" is_pre_election: true
I20260619 10:23:05.938576 31339 raft_consensus.cc:2454] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a624c6189049a1856d4ec026c966cc in term 0.
I20260619 10:23:05.938606 31077 raft_consensus.cc:2454] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a624c6189049a1856d4ec026c966cc in term 0.
I20260619 10:23:05.938899 31149 leader_election.cc:302] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47ec762b9484d1ab33a9660deb8bb8e, e3a624c6189049a1856d4ec026c966cc; no voters:
I20260619 10:23:05.939064 31421 raft_consensus.cc:2781] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:23:05.939106 31421 raft_consensus.cc:482] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:23:05.939127 31421 raft_consensus.cc:3037] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:05.940037 31421 raft_consensus.cc:504] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.940142 31421 leader_election.cc:288] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [CANDIDATE]: Term 1 election: Requested vote from peers cd7c085aea234bf487195f81e99ca6c1 (127.19.65.131:42775), a47ec762b9484d1ab33a9660deb8bb8e (127.19.65.129:42401)
I20260619 10:23:05.940457 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c963acf30e2149e2bd7290e1ce9288a8" candidate_uuid: "e3a624c6189049a1856d4ec026c966cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c085aea234bf487195f81e99ca6c1"
I20260619 10:23:05.940526 31339 raft_consensus.cc:3037] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:05.941371 31339 raft_consensus.cc:2454] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a624c6189049a1856d4ec026c966cc in term 1.
I20260619 10:23:05.941576 31077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c963acf30e2149e2bd7290e1ce9288a8" candidate_uuid: "e3a624c6189049a1856d4ec026c966cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47ec762b9484d1ab33a9660deb8bb8e"
I20260619 10:23:05.941634 31077 raft_consensus.cc:3037] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 0 FOLLOWER]: Advancing to term 1
W20260619 10:23:05.942714 31124 tablet.cc:2369] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:05.942901 31077 raft_consensus.cc:2454] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a624c6189049a1856d4ec026c966cc in term 1.
I20260619 10:23:05.943131 31149 leader_election.cc:302] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47ec762b9484d1ab33a9660deb8bb8e, e3a624c6189049a1856d4ec026c966cc; no voters:
I20260619 10:23:05.943272 31421 raft_consensus.cc:2781] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:05.943441 31421 raft_consensus.cc:686] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 1 LEADER]: Becoming Leader. State: Replica: e3a624c6189049a1856d4ec026c966cc, State: Running, Role: LEADER
I20260619 10:23:05.943521 31421 consensus_queue.cc:237] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } }
I20260619 10:23:05.944289 30911 catalog_manager.cc:5495] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a624c6189049a1856d4ec026c966cc (127.19.65.130). New cstate: current_term: 1 leader_uuid: "e3a624c6189049a1856d4ec026c966cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a624c6189049a1856d4ec026c966cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 43101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 } health_report { overall_health: UNKNOWN } } }
I20260619 10:23:05.995102 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:05.991764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.19.65.130:43101)
0619 10:23:05.991804 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:05.991806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:05.991848 (+ 42us) server_negotiation.cc:407] Connection header received
0619 10:23:05.991923 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:05.991924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:05.991946 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:05.992012 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:23:05.992205 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.992929 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.994004 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:05.994296 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:05.994377 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:05.994574 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:23:05.994715 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:23:05.994789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:05.995009 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 10:23:05.995029 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:23:05.997459 31077 raft_consensus.cc:1261] T c963acf30e2149e2bd7290e1ce9288a8 P a47ec762b9484d1ab33a9660deb8bb8e [term 1 FOLLOWER]: Refusing update from remote peer e3a624c6189049a1856d4ec026c966cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:05.997912 31423 consensus_queue.cc:1035] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47ec762b9484d1ab33a9660deb8bb8e" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:06.000105 31339 raft_consensus.cc:1261] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 1 FOLLOWER]: Refusing update from remote peer e3a624c6189049a1856d4ec026c966cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:06.000423 31423 consensus_queue.cc:1035] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7c085aea234bf487195f81e99ca6c1" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:06.003837 31426 mvcc.cc:205] Tried to move back new op lower bound from 7298517344242446336 to 7298517344025153536. Current Snapshot: MvccSnapshot[applied={T|T < 7298517344242446336}]
I20260619 10:23:06.006175 30911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:06.006256 30911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:06.006996 30911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594fab5b6c3a4bcbbee1bf48526d1f7d: Sending DeleteTablet for 3 replicas of tablet c963acf30e2149e2bd7290e1ce9288a8
I20260619 10:23:06.007405 31057 tablet_service.cc:1514] Processing DeleteTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:06 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35460
I20260619 10:23:06.007802 31325 tablet_service.cc:1514] Processing DeleteTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:06 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:33494
I20260619 10:23:06.008198 31194 tablet_service.cc:1514] Processing DeleteTablet for tablet c963acf30e2149e2bd7290e1ce9288a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:06 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:39678
I20260619 10:23:06.008671 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 30992
I20260619 10:23:06.013866 31429 mvcc.cc:205] Tried to move back new op lower bound from 7298517344242446336 to 7298517344025153536. Current Snapshot: MvccSnapshot[applied={T|T < 7298517344242446336}]
I20260619 10:23:06.017541 31435 tablet_replica.cc:330] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: stopping tablet replica
I20260619 10:23:06.017709 31434 tablet_replica.cc:330] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: stopping tablet replica
I20260619 10:23:06.017827 31435 raft_consensus.cc:2229] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:23:06.017964 31434 raft_consensus.cc:2229] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:23:06.018116 31435 raft_consensus.cc:2258] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:06.018220 31434 raft_consensus.cc:2258] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:06.019622 31435 ts_tablet_manager.cc:1905] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:23:06.021159 30897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:42401 (error 108)
I20260619 10:23:06.021472 31435 ts_tablet_manager.cc:1918] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:23:06.021586 31435 log.cc:1196] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/c963acf30e2149e2bd7290e1ce9288a8
I20260619 10:23:06.021883 31435 ts_tablet_manager.cc:1939] T c963acf30e2149e2bd7290e1ce9288a8 P e3a624c6189049a1856d4ec026c966cc: Deleting consensus metadata
I20260619 10:23:06.022267 30897 catalog_manager.cc:4841] TS e3a624c6189049a1856d4ec026c966cc (127.19.65.130:43101): tablet c963acf30e2149e2bd7290e1ce9288a8 (table test-table [id=82dd3bcf15a3410abefcb78ce3e32310]) successfully deleted
I20260619 10:23:06.026077 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 31129
I20260619 10:23:06.026115 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:06.025968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42401 (local address 127.0.0.1:35470)
0619 10:23:06.026017 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:06.026077 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:23:06.026854 30897 catalog_manager.cc:4584] TS a47ec762b9484d1ab33a9660deb8bb8e (127.19.65.129:42401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c963acf30e2149e2bd7290e1ce9288a8: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42401: connect: Connection refused (error 111)
I20260619 10:23:06.031692 31434 ts_tablet_manager.cc:1905] T c963acf30e2149e2bd7290e1ce9288a8 P cd7c085aea234bf487195f81e99ca6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:23:06.032306 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 31266
I20260619 10:23:06.042577 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 30879
I20260619 10:23:06.048149 19718 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 10:23:06.392256 19718 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 10:23:06.392350 19718 mini_ranger.cc:66] Stopping Ranger...
I20260619 10:23:06.736368 19718 mini_ranger.cc:68] Stopped Ranger
2026-06-19 10:23:06.736 UTC [30465] LOG: received smart shutdown request
2026-06-19 10:23:06.740 UTC [30465] LOG: background worker "logical replication launcher" (PID 30474) exited with exit code 1
2026-06-19 10:23:06.740 UTC [30469] LOG: shutting down
2026-06-19 10:23:06.897 UTC [30465] LOG: database system is shut down
2026-06-19T10:23:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43719 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:23:07 dist-test-slave-xd2q krb5kdc[31446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:23:07 dist-test-slave-xd2q krb5kdc[31446](info): set up 2 sockets
Jun 19 10:23:07 dist-test-slave-xd2q krb5kdc[31446](info): commencing operation
krb5kdc: starting...
W20260619 10:23:08.019129 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:23:08 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:23:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangeradmin_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangerlookup_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
I20260619 10:23:08.168157 19718 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskadFLwT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/postgres -l logfile start
2026-06-19 10:23:10.552 UTC [31476] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 10:23:10.552 UTC [31476] LOG: listening on IPv4 address "127.19.65.148", port 39621
2026-06-19 10:23:10.556 UTC [31476] LOG: listening on Unix socket "/tmp/.s.PGSQL.39621"
2026-06-19 10:23:10.576 UTC [31479] LOG: database system was shut down at 2026-06-19 10:23:09 UTC
2026-06-19 10:23:10.580 UTC [31476] LOG: database system is ready to accept connections
I20260619 10:23:11.531124 19718 mini_postgres.cc:96] Postgres bound to 39621
2026-06-19 10:23:11.537 UTC [31487] FATAL: database "slave" does not exist
127.19.65.148:39621 - accepting connections
I20260619 10:23:11.537498 19718 mini_ranger.cc:161] Starting Ranger...
I20260619 10:23:11.551795 19718 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 10:23:11.987432 19718 mini_ranger.cc:87] Created ranger Postgres database
I20260619 10:23:11.987509 19718 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin
2026-06-19 10:23:12,253 [I] DB FLAVOR :POSTGRES
2026-06-19 10:23:12,253 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:23:12,253 [I] Checking connection..
2026-06-19 10:23:12,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 10:23:12,476 [I] Checking connection passed.
2026-06-19 10:23:12,477 [I] --------- Verifying version history table ---------
2026-06-19 10:23:12,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:23:12,733 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:23:12,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:23:12,952 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:23:12,953 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 10:23:12,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 10:23:13.193 UTC [31551] WARNING: there is no transaction in progress
2026-06-19 10:23:13,199 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 10:23:13,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:23:13,446 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 10:23:13,446 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:23:13,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 10:23:13,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 10:23:13,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xd2q.c.gcp-upstream.internal','N') ;"
2026-06-19 10:23:14,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:23:14,428 [I] Table x_portal_user does not exist in database ranger
2026-06-19 10:23:14,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:23:14,674 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 10:23:14,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:23:14,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:23:15,140 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 10:23:15,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 10:23:16.006 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.021 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.035 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.049 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.063 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.079 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.129 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.135 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.148 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.156 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.166 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.178 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.187 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.193 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.199 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16.205 UTC [31659] WARNING: there is no transaction in progress
2026-06-19 10:23:16,770 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 10:23:16,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 10:23:17,006 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 10:23:17,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:23:17,249 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 10:23:17,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:23:17,503 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 10:23:17,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:23:17,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:23:17,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xd2q.c.gcp-upstream.internal';"
2026-06-19 10:23:18,179 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 10:23:18,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 10:23:18,419 [I] DB_PATCHES have already been applied
I20260619 10:23:18.424154 19718 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskadFLwT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 10:23:18.424230 19718 mini_ranger.cc:193] Using host: 127.19.65.148
I20260619 10:23:18.427156 19718 mini_ranger.cc:227] Ranger admin URL: http://127.19.65.148:54431
Jun 19, 2026 10:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:23:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:23:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 10:23:19.512928 19718 mini_ranger.cc:160] Time spent starting Ranger: real 7.975s user 0.000s sys 0.008s
Jun 19, 2026 10:23:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:23:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 10:23:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:23:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:23:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 10:23:40.944396 19718 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.65.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.65.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.65.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/rangerkms_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.65.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.65.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.65.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/HTTP_127.19.65.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 10:23:41 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 10:23:41.024060 19718 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 10:23:41.032930 19718 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 10:23:41.818420 19718 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 10:23:41.818499 19718 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms
I20260619 10:23:41.818517 19718 mini_ranger_kms.cc:227] Using postgres at 127.19.65.148:39621
2026-06-19 10:23:41,973 [I] DB FLAVOR :POSTGRES
2026-06-19 10:23:41,973 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:23:41,973 [I] Checking connection
2026-06-19 10:23:41,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 10:23:42,193 [I] connection success
2026-06-19 10:23:42,194 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 10:23:42,194 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 10:23:42,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 10:23:42,416 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 10:23:42,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:23:42,417 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 10:23:42,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.65.148:39621/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 10:23:42,679 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 10:23:43.124868 19718 mini_ranger_kms.cc:310] Created kms service
I20260619 10:23:43.299939 19718 mini_ranger_kms.cc:326] Created kudu user
I20260619 10:23:43.340467 19718 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 10:23:43 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:43.689097 19718 mini_ranger_kms.cc:384] Added ranger policy
I20260619 10:23:43.689219 19718 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms:/tmp/dist-test-taskadFLwT/build/release/bin/postgresql.jar:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskadFLwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 10:23:43.689237 19718 mini_ranger_kms.cc:240] Using host: 127.19.65.148
I20260619 10:23:43.692924 19718 mini_ranger_kms.cc:276] Ranger KMS PID: 31839
I20260619 10:23:43.693006 19718 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.65.148:39123
Jun 19, 2026 10:23:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:23:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:23:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:23:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 10:23:45 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:45.926473 19718 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.902s user 0.000s sys 0.137s
I20260619 10:23:45.926573 19718 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 10:23:45.926615 19718 mini_ranger_kms.cc:401] 127.19.65.148:39123/kms/v1/keys
Jun 19 10:23:46 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Jun 19 10:23:46 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.65.148@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
Jun 19 10:23:48 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:23:48.567633 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35625
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:35625
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:48.646231 31890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:48.646428 31890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:48.646449 31890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:48.647620 31890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:48.647660 31890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:48.647673 31890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:48.647686 31890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:48.647697 31890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:48.649111 31890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.65.148:39123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:35625
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:48.649276 31890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:48.649432 31890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:48.654023 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:48.654014 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:48.654011 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:48.654455 31890 server_base.cc:1034] running on GCE node
I20260619 10:23:48.654690 31890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:48.654976 31890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:48.656143 31890 hybrid_clock.cc:648] HybridClock initialized: now 1781864628656134 us; error 35 us; skew 500 ppm
Jun 19 10:23:48 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864628, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:48.658500 31890 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:48.658910 31890 webserver.cc:460] Webserver started at http://127.19.65.190:44993/ using document root <none> and password file <none>
I20260619 10:23:48.659106 31890 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:48.659159 31890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:48.659336 31890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:48 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864628, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:48.718305 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "ec32152fa7044140975c4b5c634704f2"
format_stamp: "Formatted at 2026-06-19 10:23:48 on dist-test-slave-xd2q"
server_key: "1d2a8f1ada5b9d552857b2741a69a96a"
server_key_iv: "4c1f5d074bdf9745c5319d2ce6b2c25e"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:48.718705 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "ec32152fa7044140975c4b5c634704f2"
format_stamp: "Formatted at 2026-06-19 10:23:48 on dist-test-slave-xd2q"
server_key: "1d2a8f1ada5b9d552857b2741a69a96a"
server_key_iv: "4c1f5d074bdf9745c5319d2ce6b2c25e"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:48.720165 31890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:23:48.734079 31905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:48.734290 31890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:23:48.734386 31890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "ec32152fa7044140975c4b5c634704f2"
format_stamp: "Formatted at 2026-06-19 10:23:48 on dist-test-slave-xd2q"
server_key: "1d2a8f1ada5b9d552857b2741a69a96a"
server_key_iv: "4c1f5d074bdf9745c5319d2ce6b2c25e"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:48.734488 31890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:48.750029 31890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:48.751983 31890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:48.752140 31890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:48.756844 31890 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:35625
I20260619 10:23:48.756880 31957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:35625 every 8 connection(s)
I20260619 10:23:48.757217 31890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:48.757966 31958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:48.760942 31958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Bootstrap starting.
I20260619 10:23:48.761624 31958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:48.761927 31958 log.cc:824] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:48.762677 31958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: No bootstrap required, opened a new log
I20260619 10:23:48.762931 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 31890
I20260619 10:23:48.763060 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:23:48.763266 19718 external_mini_cluster.cc:1428] Setting key 3700a530f071b77f027d985e30438340
I20260619 10:23:48.765609 31958 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:48.765810 31958 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:48.765882 31958 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32152fa7044140975c4b5c634704f2, State: Initialized, Role: FOLLOWER
I20260619 10:23:48.766055 31958 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:48.766144 31958 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:48.766172 31958 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:48.766239 31958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:48.766928 31958 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:48.767035 31958 leader_election.cc:302] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec32152fa7044140975c4b5c634704f2; no voters:
I20260619 10:23:48.767225 31958 leader_election.cc:288] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:23:48.767272 31963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:48.767534 31963 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 LEADER]: Becoming Leader. State: Replica: ec32152fa7044140975c4b5c634704f2, State: Running, Role: LEADER
I20260619 10:23:48.767596 31958 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:48.767658 31963 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:48.768016 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec32152fa7044140975c4b5c634704f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } } }
I20260619 10:23:48.768157 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:48.768323 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec32152fa7044140975c4b5c634704f2. Latest consensus state: current_term: 1 leader_uuid: "ec32152fa7044140975c4b5c634704f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } } }
I20260619 10:23:48.768426 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:48.768553 31972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:48.769029 31972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:48.769256 31958 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 19 10:23:48 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:48.775375 31972 catalog_manager.cc:1347] Generated new cluster ID: 16ef124ee9344d718fc9e51eff5f698c
I20260619 10:23:48.775566 31972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:23:48.779335 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:48.763783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.19.65.190:35625)
0619 10:23:48.763945 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:48.763956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:48.764048 (+ 92us) server_negotiation.cc:407] Connection header received
0619 10:23:48.765585 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:48.765592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:48.765639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:48.765776 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:48.766495 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:48.767698 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:48.768778 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:48.769358 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:48.769733 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:48.773243 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:48.773334 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:48.773334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:48.773495 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:48.776888 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:48.777327 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:48.777330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:48.777330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:48.777429 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:48.777625 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:48.777626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:48.777626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:48.777979 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:48.778487 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:48.778563 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:23:48.778997 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260619 10:23:48.787247 31972 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:23:48.788110 31972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:48.793793 31972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Generated new TSK 0
I20260619 10:23:48.793988 31972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:23:49.509614 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:49.616231 32003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:49.616422 32003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:49.616480 32003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:49.617609 32003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:49.617669 32003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:49.617712 32003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:23:49.619216 32003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:49.619490 32003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:49.619709 32003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:49.624352 32011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:49.624444 32009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:49.624352 32008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:49.625583 32003 server_base.cc:1034] running on GCE node
I20260619 10:23:49.625837 32003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:49.626123 32003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:49.627351 32003 hybrid_clock.cc:648] HybridClock initialized: now 1781864629627328 us; error 36 us; skew 500 ppm
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:49.630136 32003 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:23:49.630651 32003 webserver.cc:460] Webserver started at http://127.19.65.129:38511/ using document root <none> and password file <none>
I20260619 10:23:49.630874 32003 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:49.630928 32003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:49.631054 32003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:49.645824 32003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "e85885edd3ca4aec8af534c1333fb055"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "68c00b525f625ee87572349ea55b87e0"
server_key_iv: "413ba0a5042a0858a5753e0079a2c86d"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.646332 32003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "e85885edd3ca4aec8af534c1333fb055"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "68c00b525f625ee87572349ea55b87e0"
server_key_iv: "413ba0a5042a0858a5753e0079a2c86d"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.651471 32003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260619 10:23:49.663759 32020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.663988 32003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:23:49.664093 32003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "e85885edd3ca4aec8af534c1333fb055"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "68c00b525f625ee87572349ea55b87e0"
server_key_iv: "413ba0a5042a0858a5753e0079a2c86d"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.664196 32003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:49.679777 32003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:49.682132 32003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:49.682291 32003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:49.682601 32003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:49.682965 32003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:49.683019 32003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.683081 32003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:49.683120 32003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.690328 32003 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42815
I20260619 10:23:49.690682 32003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:23:49.691566 32133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:42815 every 8 connection(s)
I20260619 10:23:49.698911 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32003
I20260619 10:23:49.699007 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:23:49.699179 19718 external_mini_cluster.cc:1428] Setting key 42ea2178754874c25f581eb48f71adca
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
I20260619 10:23:49.708835 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:49.691463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:32843 (local address 127.19.65.190:35625)
0619 10:23:49.698209 (+ 6746us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:49.698211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:49.698220 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:49.698246 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:49.698247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:49.698260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:49.698324 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:49.699915 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.700683 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.702135 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.702461 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.702527 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:49.705061 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:49.705062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:49.705063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:49.705070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:49.706849 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:49.707346 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:49.707347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:49.707347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:49.707426 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:49.707700 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:49.707701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:49.707701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:49.707828 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:49.707880 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:49.708662 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 10:23:49.708740 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6701,"thread_start_us":79,"threads_started":1}
I20260619 10:23:49.709056 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:49.691461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.129:32843)
0619 10:23:49.697294 (+ 5833us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:49.697343 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:49.697966 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:49.698476 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:49.698481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:49.699534 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:49.699800 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:49.699806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.700783 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.700784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:49.702039 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:49.702042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.702615 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.702616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:49.702744 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:49.703338 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:49.703347 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:49.704944 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:49.706942 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:49.706948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:49.706950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:49.707218 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:49.707501 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:49.707502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:49.707503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:49.707630 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:49.707903 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:49.707909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:49.708068 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 10:23:49.708203 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 10:23:49.708345 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5770,"thread_start_us":76,"threads_started":1}
I20260619 10:23:49.709609 32134 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:49.709745 32134 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:49.709960 32134 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:49.710495 31922 ts_manager.cc:194] Registered new tserver with Master: e85885edd3ca4aec8af534c1333fb055 (127.19.65.129:42815)
I20260619 10:23:49.711143 31922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:32843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:23:49.726773 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:49.823868 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:49.824154 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:49.824239 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:49.825492 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:49.825606 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:49.825701 32140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:23:49.827129 32140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:49.827409 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:49.827618 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:49.832820 32147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:49.832845 32148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:49.832940 32140 server_base.cc:1034] running on GCE node
W20260619 10:23:49.833135 32150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:49.833391 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:49.833664 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:49.834810 32140 hybrid_clock.cc:648] HybridClock initialized: now 1781864629834796 us; error 29 us; skew 500 ppm
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:49.837337 32140 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:23:49.837890 32140 webserver.cc:460] Webserver started at http://127.19.65.130:33977/ using document root <none> and password file <none>
I20260619 10:23:49.838140 32140 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:49.838204 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:49.838328 32140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:49.851636 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "c662fdeb03724e1b884a70eb0c6bc0e7"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "ea2628092ad34796930e99987631f340"
server_key_iv: "e5b9af080f4823d8e051c9cecf2c8f0a"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.852146 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "c662fdeb03724e1b884a70eb0c6bc0e7"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "ea2628092ad34796930e99987631f340"
server_key_iv: "e5b9af080f4823d8e051c9cecf2c8f0a"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.863758 32140 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260619 10:23:49.879426 32157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.879663 32140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 10:23:49.879748 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "c662fdeb03724e1b884a70eb0c6bc0e7"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "ea2628092ad34796930e99987631f340"
server_key_iv: "e5b9af080f4823d8e051c9cecf2c8f0a"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:49.879823 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:49.913539 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:49.916002 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:49.916195 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:49.916450 32140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:49.916851 32140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:49.916936 32140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.917003 32140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:49.917057 32140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:49.925302 32140 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41369
I20260619 10:23:49.925431 32271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:41369 every 8 connection(s)
I20260619 10:23:49.925773 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:23:49.928403 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32140
I20260619 10:23:49.928508 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:23:49.928710 19718 external_mini_cluster.cc:1428] Setting key c00c022300f96dbcb924b3b25c1bd96a
Jun 19 10:23:49 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864629, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:49.938530 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:49.927599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57393 (local address 127.19.65.190:35625)
0619 10:23:49.927651 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:49.927653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:49.927958 (+ 305us) server_negotiation.cc:407] Connection header received
0619 10:23:49.928571 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:49.928573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:49.928587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:49.928616 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:49.929586 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.930330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.931865 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.932174 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.932305 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:49.934963 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:49.934964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:49.934965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:49.934972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:49.936726 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:49.937246 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:49.937247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:49.937247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:49.937332 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:49.937669 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:49.937670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:49.937670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:49.937845 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:49.937896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:49.938332 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 10:23:49.938432 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:23:49.939049 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:49.927579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.130:57393)
0619 10:23:49.927818 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:49.927855 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:49.928448 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:49.928792 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:49.928796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:49.929121 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:49.929483 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:49.929489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.930441 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.930442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:49.931767 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:49.931770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:49.932392 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:49.932393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:49.932542 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:49.933161 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:49.933170 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:49.934856 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:49.936827 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:49.936833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:49.936836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:49.937137 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:49.937423 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:49.937424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:49.937424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:49.937571 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:49.937967 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:49.937974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:49.938126 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 10:23:49.938667 (+ 541us) client_negotiation.cc:240] Negotiation successful
0619 10:23:49.938823 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260619 10:23:49.939702 32272 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:49.939788 32272 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:49.939949 32272 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:49.940361 31922 ts_manager.cc:194] Registered new tserver with Master: c662fdeb03724e1b884a70eb0c6bc0e7 (127.19.65.130:41369)
I20260619 10:23:49.940791 31922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:57393
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:23:49.960414 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:50.071564 32277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:50.071928 32277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:50.072019 32277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:50.073695 32277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:50.073767 32277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:50.073814 32277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:23:50.076030 32277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:50.076284 32277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:50.076596 32277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:50.082990 32283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.083034 32285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.082978 32282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:50.083348 32277 server_base.cc:1034] running on GCE node
I20260619 10:23:50.083541 32277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:50.083791 32277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:50.084946 32277 hybrid_clock.cc:648] HybridClock initialized: now 1781864630084928 us; error 35 us; skew 500 ppm
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:50.087412 32277 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:23:50.087841 32277 webserver.cc:460] Webserver started at http://127.19.65.131:40985/ using document root <none> and password file <none>
I20260619 10:23:50.088069 32277 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:50.088112 32277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:50.088236 32277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:50.103848 32277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "6c7647a4a2ab4ca9b4349191366aeb1b"
format_stamp: "Formatted at 2026-06-19 10:23:50 on dist-test-slave-xd2q"
server_key: "acd7f5ca5c62b300abd4de84d22188fe"
server_key_iv: "7000516ae8292978a2c1d09f453e091c"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.104561 32277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "6c7647a4a2ab4ca9b4349191366aeb1b"
format_stamp: "Formatted at 2026-06-19 10:23:50 on dist-test-slave-xd2q"
server_key: "acd7f5ca5c62b300abd4de84d22188fe"
server_key_iv: "7000516ae8292978a2c1d09f453e091c"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.106983 32277 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 10:23:50.136919 32292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.137113 32277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:23:50.137204 32277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "6c7647a4a2ab4ca9b4349191366aeb1b"
format_stamp: "Formatted at 2026-06-19 10:23:50 on dist-test-slave-xd2q"
server_key: "acd7f5ca5c62b300abd4de84d22188fe"
server_key_iv: "7000516ae8292978a2c1d09f453e091c"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.137331 32277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:50.159631 32277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:50.162001 32277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:50.162158 32277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:50.162392 32277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:50.162755 32277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:50.162806 32277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.162840 32277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:50.162870 32277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.173580 32277 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:44687
I20260619 10:23:50.174147 32277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:23:50.174434 32405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:44687 every 8 connection(s)
I20260619 10:23:50.177282 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32277
I20260619 10:23:50.177378 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:23:50.177573 19718 external_mini_cluster.cc:1428] Setting key 86fddfe07648992a81fef4aef80ba2d4
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:50.187475 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.175932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:51481 (local address 127.19.65.190:35625)
0619 10:23:50.175976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:50.175978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:50.176237 (+ 259us) server_negotiation.cc:407] Connection header received
0619 10:23:50.177054 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:50.177055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:50.177068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:50.177102 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:50.178835 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.179621 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.181130 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.181413 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.181491 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.184047 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:50.184049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:50.184050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:50.184058 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:50.185863 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.186367 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.186368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.186368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.186428 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.186722 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.186723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.186723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.186856 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:50.186912 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:50.187281 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 10:23:50.187381 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:23:50.188154 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.175919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.131:51481)
0619 10:23:50.176110 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:50.176155 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:50.176897 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:50.178221 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:50.178226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:50.178461 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:50.178722 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.178728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.179754 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.179756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.180990 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.180993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.181522 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.181523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.181662 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.182286 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:50.182296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:50.183957 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:50.185960 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.185966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.185968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.186267 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.186504 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.186505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.186505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.186634 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.186961 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:50.186966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:50.187125 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 10:23:50.187781 (+ 656us) client_negotiation.cc:240] Negotiation successful
0619 10:23:50.187925 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260619 10:23:50.189009 32406 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:50.189181 32406 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.189421 32406 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:50.189802 31922 ts_manager.cc:194] Registered new tserver with Master: 6c7647a4a2ab4ca9b4349191366aeb1b (127.19.65.131:44687)
I20260619 10:23:50.190256 31922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:51481
I20260619 10:23:50.199882 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:23:50.200544 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32003
I20260619 10:23:50.205533 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32140
I20260619 10:23:50.211504 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32277
I20260619 10:23:50.217044 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 31890
I20260619 10:23:50.222000 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35625
--webserver_interface=127.19.65.190
--webserver_port=44993
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:35625
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:50.299674 32409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:50.299880 32409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:50.299916 32409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:50.301045 32409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:50.301115 32409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:50.301138 32409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:50.301167 32409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:50.301188 32409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:50.302690 32409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:35625
--ranger_config_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=44993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:50.302940 32409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:50.303158 32409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:50.307997 32414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.307955 32415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.307966 32417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:50.308286 32409 server_base.cc:1034] running on GCE node
I20260619 10:23:50.308516 32409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:50.308776 32409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:50.309934 32409 hybrid_clock.cc:648] HybridClock initialized: now 1781864630309920 us; error 25 us; skew 500 ppm
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:50.312410 32409 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:50.312835 32409 webserver.cc:460] Webserver started at http://127.19.65.190:44993/ using document root <none> and password file <none>
I20260619 10:23:50.313045 32409 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:50.313114 32409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:50.386754 32409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:50.387535 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.387722 32409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:50.387817 32409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "ec32152fa7044140975c4b5c634704f2"
format_stamp: "Formatted at 2026-06-19 10:23:48 on dist-test-slave-xd2q"
server_key: "1d2a8f1ada5b9d552857b2741a69a96a"
server_key_iv: "4c1f5d074bdf9745c5319d2ce6b2c25e"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.388217 32409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:50.398844 32409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:50.400424 32409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:50.400583 32409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:50.404603 32409 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:35625
I20260619 10:23:50.404663 32477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:35625 every 8 connection(s)
I20260619 10:23:50.404973 32409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:50.406073 32478 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:23:50.406685 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Bootstrap starting.
I20260619 10:23:50.407485 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32409
I20260619 10:23:50.407789 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:42815
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:35625
--webserver_port=38511
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:23:50.414994 32478 log.cc:824] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:50.416687 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:23:50.416966 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Bootstrap complete.
I20260619 10:23:50.419396 32478 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:50.419631 32478 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32152fa7044140975c4b5c634704f2, State: Initialized, Role: FOLLOWER
I20260619 10:23:50.419790 32478 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:50.419852 32478 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:50.419878 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:50.419942 32478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:23:50.420854 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:50.420975 32478 leader_election.cc:302] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec32152fa7044140975c4b5c634704f2; no voters:
I20260619 10:23:50.421155 32478 leader_election.cc:288] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:23:50.421185 32482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:23:50.421294 32482 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [term 2 LEADER]: Becoming Leader. State: Replica: ec32152fa7044140975c4b5c634704f2, State: Running, Role: LEADER
I20260619 10:23:50.421382 32482 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } }
I20260619 10:23:50.421404 32478 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:50.421672 32483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec32152fa7044140975c4b5c634704f2. Latest consensus state: current_term: 2 leader_uuid: "ec32152fa7044140975c4b5c634704f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } } }
I20260619 10:23:50.421864 32483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:50.421803 32484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec32152fa7044140975c4b5c634704f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32152fa7044140975c4b5c634704f2" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 35625 } } }
I20260619 10:23:50.422077 32484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:50.422217 32490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:50.422780 32490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:50.422904 32490 catalog_manager.cc:1259] Loaded cluster ID: 16ef124ee9344d718fc9e51eff5f698c
I20260619 10:23:50.422937 32490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:23:50.423141 32478 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 10:23:50.426535 32490 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:50.427074 32490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Loaded TSK: 0
I20260619 10:23:50.427373 32490 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:23:50.517853 32480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:50.518100 32480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:50.518131 32480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:50.519767 32480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:50.519814 32480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:50.519850 32480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:23:50.522099 32480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:42815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=38511
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:50.522330 32480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:50.522564 32480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:50.528390 32518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.528908 32515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.529196 32516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:50.529695 32480 server_base.cc:1034] running on GCE node
I20260619 10:23:50.529925 32480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:50.530180 32480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:50.531363 32480 hybrid_clock.cc:648] HybridClock initialized: now 1781864630531336 us; error 40 us; skew 500 ppm
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:50.534559 32480 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:23:50.535053 32480 webserver.cc:460] Webserver started at http://127.19.65.129:38511/ using document root <none> and password file <none>
I20260619 10:23:50.535337 32480 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:50.535414 32480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:50.548245 32480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:23:50.550374 32526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.550523 32480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260619 10:23:50.550627 32480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "e85885edd3ca4aec8af534c1333fb055"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "68c00b525f625ee87572349ea55b87e0"
server_key_iv: "413ba0a5042a0858a5753e0079a2c86d"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.551061 32480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:50.572919 32480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:50.575129 32480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:50.575297 32480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:50.575533 32480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:50.575887 32480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:50.575946 32480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.575978 32480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:50.576009 32480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.583030 32480 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:42815
I20260619 10:23:50.583427 32639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:42815 every 8 connection(s)
I20260619 10:23:50.583477 32480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:23:50.585295 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32480
I20260619 10:23:50.585621 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:41369
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:35625
--webserver_port=33977
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:50.597519 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.584389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.129:44707)
0619 10:23:50.584569 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:50.584615 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:50.585257 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:50.586309 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:50.586314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:50.586557 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:50.586819 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.586825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.588460 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.588462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.589726 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.589729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.590804 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.590805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.590943 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.591532 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:50.591542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:50.593251 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:50.595610 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.595616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.595617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.595973 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.596276 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.596277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.596277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.596420 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.596919 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:50.596924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:50.597071 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 10:23:50.597162 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 10:23:50.597295 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260619 10:23:50.598336 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.584730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:44707 (local address 127.19.65.190:35625)
0619 10:23:50.584914 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:50.584922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:50.584960 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:23:50.585485 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:50.585492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:50.585545 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:50.585705 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:50.586980 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.588354 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.589847 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.590532 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.590703 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.593386 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:50.593391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:50.593391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:50.593403 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:50.595515 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.596073 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.596076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.596077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.596185 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.596504 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.596505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.596505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.596815 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:50.596891 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:50.598039 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0619 10:23:50.598149 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260619 10:23:50.599189 32640 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:50.599268 32640 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.599542 32640 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.600268 32640 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:35625 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:23:50.600337 32640 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.600389 32640 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:50.600649 32640 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.600706 32640 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.600912 32640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:23:50.690838 32644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:50.691082 32644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:50.691118 32644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:50.692906 32644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:50.692955 32644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:50.692981 32644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:23:50.694489 32644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:41369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=33977
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:50.694744 32644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:50.694989 32644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:50.700471 32650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.700860 32649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.703275 32652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:50.705180 32644 server_base.cc:1034] running on GCE node
I20260619 10:23:50.705437 32644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:50.705713 32644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:50.706940 32644 hybrid_clock.cc:648] HybridClock initialized: now 1781864630706915 us; error 39 us; skew 500 ppm
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:50.709998 32644 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:23:50.710515 32644 webserver.cc:460] Webserver started at http://127.19.65.130:33977/ using document root <none> and password file <none>
I20260619 10:23:50.710753 32644 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:50.710843 32644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:50.731751 32644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 10:23:50.732816 32660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.732961 32644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:50.733031 32644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "c662fdeb03724e1b884a70eb0c6bc0e7"
format_stamp: "Formatted at 2026-06-19 10:23:49 on dist-test-slave-xd2q"
server_key: "ea2628092ad34796930e99987631f340"
server_key_iv: "e5b9af080f4823d8e051c9cecf2c8f0a"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.733395 32644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:50.751907 32644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:50.754177 32644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:50.754393 32644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:50.754633 32644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:50.754992 32644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:50.755038 32644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.755066 32644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:50.755091 32644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.762342 32644 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41369
I20260619 10:23:50.762428 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:41369 every 8 connection(s)
I20260619 10:23:50.762719 32644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:50.772086 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 32644
I20260619 10:23:50.772336 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.65.148:39123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:44687
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:35625
--webserver_port=40985
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:23:50.772683 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.763440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60321 (local address 127.19.65.190:35625)
0619 10:23:50.763507 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:50.763529 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:50.763770 (+ 241us) server_negotiation.cc:407] Connection header received
0619 10:23:50.764407 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:50.764408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:50.764419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:50.764468 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:50.765148 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.765603 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.767075 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.767306 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.767366 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.769589 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:50.769590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:50.769591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:50.769597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:50.771190 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.771669 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.771670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.771670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.771749 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.771989 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.771990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.771990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.772119 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:50.772186 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:50.772517 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 10:23:50.772595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:23:50.773190 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.763427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.130:60321)
0619 10:23:50.763631 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:50.763676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:50.764299 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:50.764545 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:50.764549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:50.764787 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:50.765062 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.765069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.765695 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.765696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.766983 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.766985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.767397 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.767398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.767537 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.768127 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:50.768137 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:50.769499 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:50.771278 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.771284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.771306 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.771588 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.771816 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.771816 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.771817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.771909 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.772215 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:50.772222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:50.772362 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 10:23:50.772808 (+ 446us) client_negotiation.cc:240] Negotiation successful
0619 10:23:50.772981 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260619 10:23:50.773670 306 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:50.773761 306 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.773941 306 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.774597 306 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:35625 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:23:50.774675 306 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.774739 306 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:50.775097 306 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.775185 306 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.775511 306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:23:50.890795 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:50.891055 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:50.891090 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:50.892791 309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:50.892846 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:50.892894 309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:23:50.894973 309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.65.148:39123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:44687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=40985
--tserver_master_addrs=127.19.65.190:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:50.895198 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:50.895488 309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:50.900697 315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.900697 314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:50.900743 317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:50.900981 309 server_base.cc:1034] running on GCE node
I20260619 10:23:50.901263 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:50.901523 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:50.902705 309 hybrid_clock.cc:648] HybridClock initialized: now 1781864630902675 us; error 44 us; skew 500 ppm
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:50.905655 309 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:23:50.906189 309 webserver.cc:460] Webserver started at http://127.19.65.131:40985/ using document root <none> and password file <none>
I20260619 10:23:50.906491 309 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:50.906606 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:50.932125 309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:50.934290 326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.934525 309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 10:23:50.934626 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "6c7647a4a2ab4ca9b4349191366aeb1b"
format_stamp: "Formatted at 2026-06-19 10:23:50 on dist-test-slave-xd2q"
server_key: "acd7f5ca5c62b300abd4de84d22188fe"
server_key_iv: "7000516ae8292978a2c1d09f453e091c"
server_key_version: "kuduclusterkey@0"
I20260619 10:23:50.935081 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:50.967736 309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:50.969980 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:50.970144 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:50.970391 309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:50.970729 309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:50.970774 309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.970809 309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:50.970829 309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:50.977722 309 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:44687
I20260619 10:23:50.977808 439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:44687 every 8 connection(s)
I20260619 10:23:50.978091 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:23:50 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864630, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:50.987001 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 309
I20260619 10:23:50.987424 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.978876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:35625 (local address 127.19.65.131:58941)
0619 10:23:50.979042 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:50.979073 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:50.979614 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:50.979851 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:50.979855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:50.980010 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:50.980235 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.980239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.980891 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.980893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.981837 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:50.981839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.982243 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.982245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:50.982364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.982844 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:50.982853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:50.984170 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:50.985829 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.985835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.985837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.986126 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.986332 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:50.986333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:50.986334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:50.986476 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:50.986701 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:50.986706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:50.986858 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 10:23:50.986992 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 10:23:50.987143 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260619 10:23:50.987697 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.978900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:58941 (local address 127.19.65.190:35625)
0619 10:23:50.978953 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:50.978955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:50.979164 (+ 209us) server_negotiation.cc:407] Connection header received
0619 10:23:50.979715 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:50.979717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:50.979728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:50.979776 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:50.980320 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.980786 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.981928 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.982144 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.982219 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.984254 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:50.984255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:50.984256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:50.984262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:50.985720 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.986214 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.986215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.986216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.986266 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.986543 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.986544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.986544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.986638 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:50.986686 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:50.987027 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 10:23:50.987112 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:23:50.988144 440 heartbeater.cc:347] Connected to a master server at 127.19.65.190:35625
I20260619 10:23:50.988230 440 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.988418 440 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.989045 440 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:35625 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:23:50.989102 440 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.989157 440 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:50.989425 440 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:50.989487 440 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
W20260619 10:23:50.989832 440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:23:50.998234 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:50.987598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.19.65.190:35625)
0619 10:23:50.987765 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:50.987767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:50.987903 (+ 136us) server_negotiation.cc:407] Connection header received
0619 10:23:50.987968 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:50.987968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:50.987979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:50.988029 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:50.988758 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.989409 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.990724 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:50.991016 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:50.991084 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:50.995446 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:50.995447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:50.995448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:50.995454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:50.997120 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.997464 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.997465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.997465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.997520 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:50.997744 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:50.997744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:50.997745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:50.997848 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:50.998025 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:50.998061 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 10:23:50.998130 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":52,"threads_started":1}
Jun 19 10:23:51 dist-test-slave-xd2q krb5kdc[31446](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864631, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:51.601073 32640 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:51.601207 32640 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:51.601645 32442 ts_manager.cc:194] Registered new tserver with Master: e85885edd3ca4aec8af534c1333fb055 (127.19.65.129:42815)
I20260619 10:23:51.602345 32442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:44707
I20260619 10:23:51.775660 306 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:51.775794 306 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:51.776125 32442 ts_manager.cc:194] Registered new tserver with Master: c662fdeb03724e1b884a70eb0c6bc0e7 (127.19.65.130:41369)
I20260619 10:23:51.776656 32442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:60321
Jun 19 10:23:51 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864631, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
I20260619 10:23:51.989993 440 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:51.990140 440 heartbeater.cc:511] Master 127.19.65.190:35625 requested a full tablet report, sending...
I20260619 10:23:51.990434 32442 ts_manager.cc:194] Registered new tserver with Master: 6c7647a4a2ab4ca9b4349191366aeb1b (127.19.65.131:44687)
I20260619 10:23:51.990824 32442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:58941
I20260619 10:23:52.000169 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:23:52.007478 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.001132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.19.65.190:35625)
0619 10:23:52.001243 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.001246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.001298 (+ 52us) server_negotiation.cc:407] Connection header received
0619 10:23:52.001367 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.001368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.001381 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.001443 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:52.001990 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.002490 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.003186 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.003468 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.003550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.004769 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:52.004771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:52.004771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:52.004795 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:52.006295 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:52.006657 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:52.006658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:52.006659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:52.006717 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:52.006915 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:52.006916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:52.006917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:52.007042 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:52.007141 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.007241 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.007350 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260619 10:23:52.009413 32442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:23:52.011016 32442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:23:52.018369 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.013660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:44687 (local address 127.0.0.1:38744)
0619 10:23:52.014103 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.014144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:52.014205 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:52.014473 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:52.014476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:52.014478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:52.014534 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.014536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.015810 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.015810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.017136 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.017138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.018085 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.018086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.018159 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.018170 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 10:23:52.018235 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:23:52.018286 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":76,"threads_started":1}
I20260619 10:23:52.018669 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.013773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.19.65.130:41369)
0619 10:23:52.013913 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.013917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.014385 (+ 468us) server_negotiation.cc:407] Connection header received
0619 10:23:52.014509 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.014512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.014528 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.014578 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:52.015063 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.016093 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.017550 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.018238 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.018277 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.018322 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.018549 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.018588 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260619 10:23:52.018975 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.014067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.19.65.131:44687)
0619 10:23:52.014208 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.014213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.014222 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:52.014290 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.014292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.014308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.014382 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:52.014614 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.015723 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.017221 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.017929 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.018000 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.018051 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.018833 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.018891 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260619 10:23:52.019225 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.013661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41369 (local address 127.0.0.1:56766)
0619 10:23:52.014319 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.014329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:52.014413 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:52.014913 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:52.014914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:52.014916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:52.014969 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.014971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.016182 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.016182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.017466 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.017468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.018392 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.018392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.018464 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.018468 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:23:52.019122 (+ 654us) client_negotiation.cc:240] Negotiation successful
0619 10:23:52.019160 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":7,"thread_start_us":72,"threads_started":1}
I20260619 10:23:52.019626 374 tablet_service.cc:1467] Processing CreateTablet for tablet c5b91678ced44647ad3188a9676c9b1c (DEFAULT_TABLE table=test-table [id=cfa6c075fa91498eb9212ddbeba4f545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:52.019917 32708 tablet_service.cc:1467] Processing CreateTablet for tablet c5b91678ced44647ad3188a9676c9b1c (DEFAULT_TABLE table=test-table [id=cfa6c075fa91498eb9212ddbeba4f545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:52.019994 374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b91678ced44647ad3188a9676c9b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:52.020221 32708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b91678ced44647ad3188a9676c9b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:52.024272 473 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Bootstrap starting.
I20260619 10:23:52.025398 473 tablet_bootstrap.cc:654] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:52.025820 473 log.cc:824] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:52.027906 473 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: No bootstrap required, opened a new log
I20260619 10:23:52.028025 473 ts_tablet_manager.cc:1397] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:23:52.030603 473 raft_consensus.cc:348] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.030982 473 raft_consensus.cc:374] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:52.031054 473 raft_consensus.cc:729] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c662fdeb03724e1b884a70eb0c6bc0e7, State: Initialized, Role: FOLLOWER
I20260619 10:23:52.031268 473 consensus_queue.cc:260] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.031591 473 ts_tablet_manager.cc:1428] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:23:52.031814 306 heartbeater.cc:503] Master 127.19.65.190:35625 was elected leader, sending a full tablet report...
I20260619 10:23:52.035444 474 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Bootstrap starting.
I20260619 10:23:52.036057 474 tablet_bootstrap.cc:654] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:52.036363 474 log.cc:824] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:52.037072 474 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: No bootstrap required, opened a new log
I20260619 10:23:52.037143 474 ts_tablet_manager.cc:1397] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:23:52.039628 474 raft_consensus.cc:348] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.039789 474 raft_consensus.cc:374] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:52.039829 474 raft_consensus.cc:729] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7647a4a2ab4ca9b4349191366aeb1b, State: Initialized, Role: FOLLOWER
I20260619 10:23:52.039939 474 consensus_queue.cc:260] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.040174 474 ts_tablet_manager.cc:1428] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:23:52.040191 440 heartbeater.cc:503] Master 127.19.65.190:35625 was elected leader, sending a full tablet report...
I20260619 10:23:52.042678 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.013872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42815 (local address 127.0.0.1:53990)
0619 10:23:52.018720 (+ 4848us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.018729 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:52.018760 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:52.039634 (+ 20874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:52.039635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:52.039638 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:52.039687 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.039689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.040922 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.040923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.041609 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.041611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.042447 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.042448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.042505 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.042509 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:23:52.042564 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:23:52.042605 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4838,"thread_start_us":60,"threads_started":1}
I20260619 10:23:52.042690 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.013964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.19.65.129:42815)
0619 10:23:52.039434 (+ 25470us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.039442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.039452 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:52.039481 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.039483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.039527 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.039653 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:52.039775 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.040823 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.041699 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.042360 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.042449 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.042503 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.042586 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.042621 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25434,"thread_start_us":57,"threads_started":1}
I20260619 10:23:52.043537 32573 tablet_service.cc:1467] Processing CreateTablet for tablet c5b91678ced44647ad3188a9676c9b1c (DEFAULT_TABLE table=test-table [id=cfa6c075fa91498eb9212ddbeba4f545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:52.043824 32573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b91678ced44647ad3188a9676c9b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:52.046330 479 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Bootstrap starting.
I20260619 10:23:52.046938 479 tablet_bootstrap.cc:654] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:52.047220 479 log.cc:824] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:52.047933 479 tablet_bootstrap.cc:492] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: No bootstrap required, opened a new log
I20260619 10:23:52.048012 479 ts_tablet_manager.cc:1397] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:23:52.049896 479 raft_consensus.cc:348] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.050037 479 raft_consensus.cc:374] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:52.050077 479 raft_consensus.cc:729] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85885edd3ca4aec8af534c1333fb055, State: Initialized, Role: FOLLOWER
I20260619 10:23:52.050230 479 consensus_queue.cc:260] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.050468 479 ts_tablet_manager.cc:1428] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:23:52.050475 32640 heartbeater.cc:503] Master 127.19.65.190:35625 was elected leader, sending a full tablet report...
W20260619 10:23:52.094231 32641 tablet.cc:2369] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 10:23:52 dist-test-slave-xd2q krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864631, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for HTTP/127.19.65.148@KRBTEST.COM
W20260619 10:23:52.229358 441 tablet.cc:2369] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:52.246708 478 raft_consensus.cc:482] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:23:52.246873 478 raft_consensus.cc:504] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.248010 478 leader_election.cc:288] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e85885edd3ca4aec8af534c1333fb055 (127.19.65.129:42815), c662fdeb03724e1b884a70eb0c6bc0e7 (127.19.65.130:41369)
I20260619 10:23:52.257982 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.247837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:41609 (local address 127.19.65.130:41369)
0619 10:23:52.247917 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.247919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.254372 (+ 6453us) server_negotiation.cc:407] Connection header received
0619 10:23:52.254480 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.254481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.254491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.254519 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:52.254716 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.255476 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.256990 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.257592 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.257638 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.257646 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.257874 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.257913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 10:23:52.258225 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.247747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41369 (local address 127.19.65.131:41609)
0619 10:23:52.254277 (+ 6530us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.254289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:52.254403 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:52.254580 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:52.254581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:52.254582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:52.254647 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.254648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.255592 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.255593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.256886 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.256888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.257716 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.257717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.257785 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.257789 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:23:52.258097 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 10:23:52.258135 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6504,"thread_start_us":69,"threads_started":1}
I20260619 10:23:52.258610 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b91678ced44647ad3188a9676c9b1c" candidate_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" is_pre_election: true
I20260619 10:23:52.258785 32727 raft_consensus.cc:2454] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c7647a4a2ab4ca9b4349191366aeb1b in term 0.
I20260619 10:23:52.259023 329 leader_election.cc:302] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7647a4a2ab4ca9b4349191366aeb1b, c662fdeb03724e1b884a70eb0c6bc0e7; no voters:
I20260619 10:23:52.259150 478 raft_consensus.cc:2781] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:23:52.259198 478 raft_consensus.cc:482] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:23:52.259227 478 raft_consensus.cc:3037] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:52.260236 478 raft_consensus.cc:504] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.260386 478 leader_election.cc:288] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [CANDIDATE]: Term 1 election: Requested vote from peers e85885edd3ca4aec8af534c1333fb055 (127.19.65.129:42815), c662fdeb03724e1b884a70eb0c6bc0e7 (127.19.65.130:41369)
I20260619 10:23:52.260579 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b91678ced44647ad3188a9676c9b1c" candidate_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7"
I20260619 10:23:52.260660 32727 raft_consensus.cc:3037] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:52.261484 32727 raft_consensus.cc:2454] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c7647a4a2ab4ca9b4349191366aeb1b in term 1.
I20260619 10:23:52.261647 329 leader_election.cc:302] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7647a4a2ab4ca9b4349191366aeb1b, c662fdeb03724e1b884a70eb0c6bc0e7; no voters:
I20260619 10:23:52.261723 478 raft_consensus.cc:2781] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:52.261852 478 raft_consensus.cc:686] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 1 LEADER]: Becoming Leader. State: Replica: 6c7647a4a2ab4ca9b4349191366aeb1b, State: Running, Role: LEADER
I20260619 10:23:52.261945 478 consensus_queue.cc:237] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } }
I20260619 10:23:52.262755 32442 catalog_manager.cc:5495] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c7647a4a2ab4ca9b4349191366aeb1b (127.19.65.131). New cstate: current_term: 1 leader_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 } health_report { overall_health: UNKNOWN } } }
W20260619 10:23:52.264003 307 tablet.cc:2369] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:52.264719 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.247975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:52019 (local address 127.19.65.129:42815)
0619 10:23:52.248019 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.248022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.254250 (+ 6228us) server_negotiation.cc:407] Connection header received
0619 10:23:52.254378 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.254380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.254391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.254422 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:52.254656 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.255453 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.263761 (+ 8308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.264324 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.264388 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.264397 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.264606 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.264651 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:23:52.264963 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.247508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42815 (local address 127.19.65.131:52019)
0619 10:23:52.254167 (+ 6659us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.254182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:52.254284 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:52.254496 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:52.254497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:52.254499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:52.254577 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.254579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.255578 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.255579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.263666 (+ 8087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:52.263668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.264456 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.264457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:52.264530 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.264533 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:23:52.264839 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 10:23:52.264882 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6637,"mutex_wait_us":1899,"thread_start_us":105,"threads_started":1}
I20260619 10:23:52.265302 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b91678ced44647ad3188a9676c9b1c" candidate_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85885edd3ca4aec8af534c1333fb055" is_pre_election: true
I20260619 10:23:52.265455 32594 raft_consensus.cc:2454] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c7647a4a2ab4ca9b4349191366aeb1b in term 0.
I20260619 10:23:52.265573 32593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b91678ced44647ad3188a9676c9b1c" candidate_uuid: "6c7647a4a2ab4ca9b4349191366aeb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85885edd3ca4aec8af534c1333fb055"
I20260619 10:23:52.265630 32593 raft_consensus.cc:3037] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:52.266433 32593 raft_consensus.cc:2454] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c7647a4a2ab4ca9b4349191366aeb1b in term 1.
I20260619 10:23:52.307367 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.303789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.19.65.131:44687)
0619 10:23:52.303867 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:52.303869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:52.303878 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:52.303917 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:52.303917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:52.303934 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:52.304011 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:23:52.304157 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.304893 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.306087 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:52.306381 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:52.306448 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:52.306774 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:23:52.306905 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:23:52.306988 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:52.307222 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 10:23:52.307243 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 10:23:52.316491 32727 raft_consensus.cc:1261] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 1 FOLLOWER]: Refusing update from remote peer 6c7647a4a2ab4ca9b4349191366aeb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:52.316852 486 consensus_queue.cc:1035] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c662fdeb03724e1b884a70eb0c6bc0e7" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:52.318727 32593 raft_consensus.cc:1261] T c5b91678ced44647ad3188a9676c9b1c P e85885edd3ca4aec8af534c1333fb055 [term 1 FOLLOWER]: Refusing update from remote peer 6c7647a4a2ab4ca9b4349191366aeb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:52.319130 486 consensus_queue.cc:1035] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85885edd3ca4aec8af534c1333fb055" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 42815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:52.323098 489 mvcc.cc:205] Tried to move back new op lower bound from 7298517533965062144 to 7298517533745512448. Current Snapshot: MvccSnapshot[applied={T|T < 7298517533965062144}]
I20260619 10:23:52.323678 492 mvcc.cc:205] Tried to move back new op lower bound from 7298517533965062144 to 7298517533745512448. Current Snapshot: MvccSnapshot[applied={T|T < 7298517533965062144}]
I20260619 10:23:52.326485 32442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:52.326577 32442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:52.327266 32442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec32152fa7044140975c4b5c634704f2: Sending DeleteTablet for 3 replicas of tablet c5b91678ced44647ad3188a9676c9b1c
I20260619 10:23:52.327768 374 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b91678ced44647ad3188a9676c9b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:52 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:38744
I20260619 10:23:52.327970 32573 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b91678ced44647ad3188a9676c9b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:52 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:53990
I20260619 10:23:52.328171 32708 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b91678ced44647ad3188a9676c9b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:52 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:56766
I20260619 10:23:52.328732 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32480
I20260619 10:23:52.332577 490 mvcc.cc:205] Tried to move back new op lower bound from 7298517533965062144 to 7298517533745512448. Current Snapshot: MvccSnapshot[applied={T|T < 7298517533965062144}]
I20260619 10:23:52.332789 499 tablet_replica.cc:330] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: stopping tablet replica
I20260619 10:23:52.332888 499 raft_consensus.cc:2229] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:23:52.332954 499 raft_consensus.cc:2258] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:52.333856 499 ts_tablet_manager.cc:1905] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:23:52.334036 497 tablet_replica.cc:330] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: stopping tablet replica
I20260619 10:23:52.334183 497 raft_consensus.cc:2229] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:23:52.334309 497 raft_consensus.cc:2258] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:52.335644 499 ts_tablet_manager.cc:1918] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:23:52.335713 499 log.cc:1196] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/c5b91678ced44647ad3188a9676c9b1c
I20260619 10:23:52.336167 499 ts_tablet_manager.cc:1939] T c5b91678ced44647ad3188a9676c9b1c P c662fdeb03724e1b884a70eb0c6bc0e7: Deleting consensus metadata
I20260619 10:23:52.336560 32428 catalog_manager.cc:4841] TS c662fdeb03724e1b884a70eb0c6bc0e7 (127.19.65.130:41369): tablet c5b91678ced44647ad3188a9676c9b1c (table test-table [id=cfa6c075fa91498eb9212ddbeba4f545]) successfully deleted
I20260619 10:23:52.336726 497 ts_tablet_manager.cc:1905] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:23:52.338622 497 ts_tablet_manager.cc:1918] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 10:23:52.340444 32427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:42815 (error 108)
I20260619 10:23:52.341157 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:52.341046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:42815 (local address 127.0.0.1:53996)
0619 10:23:52.341088 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:52.341124 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 10:23:52.341261 32427 catalog_manager.cc:4584] TS e85885edd3ca4aec8af534c1333fb055 (127.19.65.129:42815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5b91678ced44647ad3188a9676c9b1c: Network error: Client connection negotiation failed: client connection to 127.19.65.129:42815: connect: Connection refused (error 111)
I20260619 10:23:52.341440 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32644
I20260619 10:23:52.341980 497 log.cc:1196] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/c5b91678ced44647ad3188a9676c9b1c
I20260619 10:23:52.342893 497 ts_tablet_manager.cc:1939] T c5b91678ced44647ad3188a9676c9b1c P 6c7647a4a2ab4ca9b4349191366aeb1b: Deleting consensus metadata
I20260619 10:23:52.343276 32427 catalog_manager.cc:4841] TS 6c7647a4a2ab4ca9b4349191366aeb1b (127.19.65.131:44687): tablet c5b91678ced44647ad3188a9676c9b1c (table test-table [id=cfa6c075fa91498eb9212ddbeba4f545]) successfully deleted
I20260619 10:23:52.348304 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 309
I20260619 10:23:52.353808 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 32409
I20260619 10:23:52.360404 19718 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 10:23:52.704445 19718 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 10:23:52.704525 19718 mini_ranger.cc:66] Stopping Ranger...
I20260619 10:23:53.048461 19718 mini_ranger.cc:68] Stopped Ranger
2026-06-19 10:23:53.048 UTC [31476] LOG: received smart shutdown request
2026-06-19 10:23:53.052 UTC [31476] LOG: background worker "logical replication launcher" (PID 31485) exited with exit code 1
2026-06-19 10:23:53.052 UTC [31480] LOG: shutting down
2026-06-19 10:23:53.192 UTC [31476] LOG: database system is shut down
2026-06-19T10:23:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46304 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:23:53 dist-test-slave-xd2q krb5kdc[510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:23:53 dist-test-slave-xd2q krb5kdc[510](info): set up 2 sockets
Jun 19 10:23:53 dist-test-slave-xd2q krb5kdc[510](info): commencing operation
krb5kdc: starting...
W20260619 10:23:54.327080 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:23:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:23:54.436168 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46713
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:54.513962 523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:54.514199 523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:54.514235 523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:54.515447 523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:54.515503 523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:54.515527 523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:54.515555 523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:54.515585 523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:54.517055 523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:54.517313 523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:54.517526 523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:54.522269 529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:54.522310 528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:54.522313 531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:54.522348 523 server_base.cc:1034] running on GCE node
I20260619 10:23:54.522688 523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:54.522945 523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:54.524096 523 hybrid_clock.cc:648] HybridClock initialized: now 1781864634524070 us; error 45 us; skew 500 ppm
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:54.529570 523 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:54.530014 523 webserver.cc:460] Webserver started at http://127.19.65.190:43257/ using document root <none> and password file <none>
I20260619 10:23:54.530231 523 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:54.530293 523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:54.530438 523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:54.531549 523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "3bdbff64464b44bfb7d96d7905db9f86"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "013c4df1c56590d402e987b09e7339bd"
server_key_iv: "289cda8b3d15ec67a2fa79bb004d214b"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.531903 523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "3bdbff64464b44bfb7d96d7905db9f86"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "013c4df1c56590d402e987b09e7339bd"
server_key_iv: "289cda8b3d15ec67a2fa79bb004d214b"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.533218 523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:23:54.533928 538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:54.534114 523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:23:54.534210 523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "3bdbff64464b44bfb7d96d7905db9f86"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "013c4df1c56590d402e987b09e7339bd"
server_key_iv: "289cda8b3d15ec67a2fa79bb004d214b"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.534286 523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:54.549104 523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:54.551258 523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:54.551440 523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:54.555018 523 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46713
I20260619 10:23:54.555056 590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46713 every 8 connection(s)
I20260619 10:23:54.555500 523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:54.556129 591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:54.557567 591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:23:54.560302 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 523
I20260619 10:23:54.560417 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:23:54.560604 19718 external_mini_cluster.cc:1428] Setting key 2b1667dbef4fbafe28c3ad9ab4591397
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:54.567152 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.558166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46713 (local address 127.0.0.1:48850)
0619 10:23:54.558309 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.558349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.558782 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.559005 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.559006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.559219 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.559384 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.559387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.560378 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.560380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.561172 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.561175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.561911 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.561913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.562004 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.562463 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.562474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.564030 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.565748 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.565753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.565755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.565943 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.566180 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.566181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.566181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.566266 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.566593 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.566598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.566654 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.566757 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.566982 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":94,"thread_start_us":77,"threads_started":1}
I20260619 10:23:54.567160 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.558212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.19.65.190:46713)
0619 10:23:54.558354 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.558358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.558447 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:23:54.558900 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.558902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.558918 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.558999 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.559461 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.560249 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.561282 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.561798 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.561917 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.564131 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.564138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.564138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.564145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.565648 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.566036 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.566039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.566039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.566114 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.566332 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.566333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.566334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.566523 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.566611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.566782 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.566922 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260619 10:23:54.567917 591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.65.189@KRBTEST.COM; defaulting to no policy
I20260619 10:23:54.568225 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.568114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60066)
0619 10:23:54.568160 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.568199 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 10:23:54.568358 540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
I20260619 10:23:54.568751 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.568661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60080)
0619 10:23:54.568708 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.568730 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 10:23:54.568856 591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:43353: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
I20260619 10:23:54.584281 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:43353
--webserver_interface=127.19.65.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:23:54.618076 591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } attempt: 1
I20260619 10:23:54.618604 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.618460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60084)
0619 10:23:54.618536 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.618575 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 10:23:54.619024 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.618933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60086)
0619 10:23:54.618981 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.619006 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 10:23:54.619138 591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:43353: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
W20260619 10:23:54.662726 598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:54.662928 598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:54.662952 598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:54.664057 598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:54.664098 598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:54.664115 598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:54.664129 598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:54.664143 598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:54.665552 598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.65.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:54.665732 598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:54.665901 598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:54.670636 603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:54.670677 598 server_base.cc:1034] running on GCE node
W20260619 10:23:54.670684 604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:54.670639 606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:54.671067 598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:54.671334 598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:54.672516 598 hybrid_clock.cc:648] HybridClock initialized: now 1781864634672481 us; error 48 us; skew 500 ppm
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:54.675012 598 init.cc:377] Logged in from keytab as kudu/127.19.65.189@KRBTEST.COM (short username kudu)
I20260619 10:23:54.675457 598 webserver.cc:460] Webserver started at http://127.19.65.189:45429/ using document root <none> and password file <none>
I20260619 10:23:54.675645 598 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:54.675685 598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:54.675778 598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:54.676728 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data/instance:
uuid: "93e35dcad6fb4895ba3129e3a270b295"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "8ca2668e90795d57452e4be11538aefb"
server_key_iv: "ae09e0635805a2bd16c7827087d8b4a4"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.677042 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal/instance:
uuid: "93e35dcad6fb4895ba3129e3a270b295"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "8ca2668e90795d57452e4be11538aefb"
server_key_iv: "ae09e0635805a2bd16c7827087d8b4a4"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.678251 598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:23:54.678884 613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:54.679078 598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:23:54.679157 598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "93e35dcad6fb4895ba3129e3a270b295"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "8ca2668e90795d57452e4be11538aefb"
server_key_iv: "ae09e0635805a2bd16c7827087d8b4a4"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.679242 598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:54.687282 591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } attempt: 2
I20260619 10:23:54.687691 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.687574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60100)
0619 10:23:54.687631 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.687665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 10:23:54.687995 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.687905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60106)
0619 10:23:54.687949 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.687971 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 10:23:54.688130 591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:43353: Network error: Client connection negotiation failed: client connection to 127.19.65.189:43353: connect: Connection refused (error 111)
I20260619 10:23:54.693233 598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:54.694947 598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:54.695106 598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:54.698845 598 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:43353
I20260619 10:23:54.698875 665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.189:43353 every 8 connection(s)
I20260619 10:23:54.699185 598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
I20260619 10:23:54.700037 666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:54.701339 666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:54.708470 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 598
I20260619 10:23:54.708567 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-1/wal/instance
I20260619 10:23:54.708768 19718 external_mini_cluster.cc:1428] Setting key a6884ca4ba53777d6f0461cb3f1284d1
I20260619 10:23:54.709945 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.701888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46713 (local address 127.0.0.1:48864)
0619 10:23:54.702040 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.702076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.702511 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.702737 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.702740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.702967 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.703134 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.703137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.703873 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.703874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.704618 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.704620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.704972 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.704973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.705062 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.705565 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.705573 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.706878 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.708660 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.708666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.708668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.708939 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.709132 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.709133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.709133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.709221 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.709510 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.709514 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.709610 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.709683 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.709785 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260619 10:23:54.709964 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.701950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.19.65.190:46713)
0619 10:23:54.702007 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.702009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.702159 (+ 150us) server_negotiation.cc:407] Connection header received
0619 10:23:54.702598 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.702599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.702612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.702673 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.703233 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.703792 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.704699 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.704901 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.704956 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.706991 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.706993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.706993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.707000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.708534 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.709009 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.709010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.709011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.709069 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.709307 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.709308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.709309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.709450 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.709516 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.709723 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.709810 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:23:54.710510 666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.188@KRBTEST.COM; defaulting to no policy
I20260619 10:23:54.717684 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.710745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60118)
0619 10:23:54.710789 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.710799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.710838 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.711091 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.711092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.711222 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.711258 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.711259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.712136 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.712137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.712613 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.712614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.713084 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.713085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.713150 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.713539 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.713544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.714850 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.716688 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.716689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.716689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.716850 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.717062 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.717063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.717063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.717119 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.717440 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.717440 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.717480 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.717551 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.717625 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:23:54.717832 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.710826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.19.65.189:43353)
0619 10:23:54.710974 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.710977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.710991 (+ 14us) server_negotiation.cc:407] Connection header received
0619 10:23:54.711005 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.711007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.711016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.711091 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.711367 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.712034 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.712703 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.713002 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.713085 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.714942 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.714948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.714949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.714955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.716597 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.716939 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.716941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.716942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.716998 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.717198 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.717199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.717199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.717348 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.717429 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.717578 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.717672 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260619 10:23:54.718437 666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:23:54.718737 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.718617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52028)
0619 10:23:54.718668 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.718713 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.188@KRBTEST.COM" created.
W20260619 10:23:54.718884 616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
I20260619 10:23:54.719283 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.719164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52040)
0619 10:23:54.719227 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.719255 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 10:23:54.719422 666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:46153: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
I20260619 10:23:54.733103 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:46153
--webserver_interface=127.19.65.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:23:54.768648 666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } attempt: 1
I20260619 10:23:54.769208 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.769073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52054)
0619 10:23:54.769132 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.769175 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 10:23:54.769601 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.769498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52068)
0619 10:23:54.769550 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.769577 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 10:23:54.769719 666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:46153: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
I20260619 10:23:54.779275 591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } attempt: 3
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:23:54.786559 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.779616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60128)
0619 10:23:54.779678 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.779695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.779763 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.779915 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.779916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.780071 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.780135 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.780137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.780818 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.780819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.781386 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.781388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.781840 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.781841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.781925 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.782381 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.782386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.783797 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.785582 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.785582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.785583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.785750 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.785953 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.785954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.785954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.786007 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.786311 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.786312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.786371 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.786421 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.786497 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 10:23:54.786625 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.779684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.19.65.189:43353)
0619 10:23:54.779738 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.779740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.779750 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:54.779829 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.779830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.779841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.779909 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.780224 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.780715 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.781475 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.781768 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.781841 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.783887 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.783888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.783889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.783896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.785492 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.785820 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.785821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.785822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.785885 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.786098 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.786099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.786099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.786240 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.786308 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.786461 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.786537 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:23:54.787132 591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:23:54.787470 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.787383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52084)
0619 10:23:54.787422 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.787450 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:23:54.787810 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.787731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52094)
0619 10:23:54.787775 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.787796 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 10:23:54.787926 591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:46153: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
W20260619 10:23:54.812608 673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:54.812783 673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:54.812803 673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:54.813846 673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:54.813877 673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:54.813889 673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:54.813901 673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:54.813930 673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:54.815399 673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46713,127.19.65.189:43353,127.19.65.188:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.65.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:54.815579 673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:54.815723 673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:23:54.819072 591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } attempt: 1
I20260619 10:23:54.819502 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.819381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52106)
0619 10:23:54.819433 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.819463 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:23:54.820034 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.819902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52122)
0619 10:23:54.819940 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.819972 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 10:23:54.820233 591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:46153: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
W20260619 10:23:54.820263 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:54.820339 678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:54.820281 679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:54.820801 673 server_base.cc:1034] running on GCE node
I20260619 10:23:54.821038 673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:54.821323 673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:54.822484 673 hybrid_clock.cc:648] HybridClock initialized: now 1781864634822466 us; error 33 us; skew 500 ppm
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:54.824738 673 init.cc:377] Logged in from keytab as kudu/127.19.65.188@KRBTEST.COM (short username kudu)
I20260619 10:23:54.825138 673 webserver.cc:460] Webserver started at http://127.19.65.188:38821/ using document root <none> and password file <none>
I20260619 10:23:54.825297 673 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:54.825340 673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:54.825430 673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:54.826321 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data/instance:
uuid: "6c2fd9800eb7456687f55e25b15291b9"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "ac89efb514b8500c89cb4e4c76914a54"
server_key_iv: "98bb82bd390b791da783e2b0abe24588"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.826607 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal/instance:
uuid: "6c2fd9800eb7456687f55e25b15291b9"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "ac89efb514b8500c89cb4e4c76914a54"
server_key_iv: "98bb82bd390b791da783e2b0abe24588"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.827930 673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:23:54.828598 688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:54.828802 673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:54.828868 673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "6c2fd9800eb7456687f55e25b15291b9"
format_stamp: "Formatted at 2026-06-19 10:23:54 on dist-test-slave-xd2q"
server_key: "ac89efb514b8500c89cb4e4c76914a54"
server_key_iv: "98bb82bd390b791da783e2b0abe24588"
server_key_version: "encryptionkey@0"
I20260619 10:23:54.828953 673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:54.838263 666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } attempt: 2
I20260619 10:23:54.838629 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.838505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52126)
0619 10:23:54.838547 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.838593 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:23:54.839659 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.839546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52128)
0619 10:23:54.839586 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.839633 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 10:23:54.839887 666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:46153: Network error: Client connection negotiation failed: client connection to 127.19.65.188:46153: connect: Connection refused (error 111)
I20260619 10:23:54.854565 673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:54.856570 673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:54.856722 673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:54.860689 673 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:46153
I20260619 10:23:54.860738 740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.188:46153 every 8 connection(s)
I20260619 10:23:54.861088 673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
I20260619 10:23:54.861882 741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:54.863426 741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:23:54.867479 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 673
I20260619 10:23:54.867583 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864489541676-19718-0/minicluster-data/master-2/wal/instance
I20260619 10:23:54.867797 19718 external_mini_cluster.cc:1428] Setting key 86a3c59f3e927a26a3e164665cbb607e
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:54.872277 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.864098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.19.65.190:46713)
0619 10:23:54.864158 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.864162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.864376 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:23:54.864778 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.864780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.864788 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.864832 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.865441 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.865892 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.866792 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.866994 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.867063 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.869520 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.869521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.869522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.869528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.870891 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.871263 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.871264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.871264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.871363 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.871651 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.871652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.871652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.871785 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.871849 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.872153 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.872215 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:23:54.872527 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.864078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46713 (local address 127.0.0.1:48870)
0619 10:23:54.864253 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.864284 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.864708 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.864907 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.864910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.865181 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.865364 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.865368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.865968 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.865968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.866722 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.866724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.867077 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.867078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.867173 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.867777 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.867794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.869435 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.871000 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.871005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.871007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.871197 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.871428 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.871428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.871429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.871559 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.871848 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.871854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.872014 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.872130 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.872275 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260619 10:23:54.873162 741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:23:54.876751 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.868166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.19.65.188:46153)
0619 10:23:54.868310 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.868314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.868377 (+ 63us) server_negotiation.cc:407] Connection header received
0619 10:23:54.868406 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.868408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.868429 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.868504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.869204 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.870141 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.870951 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.871334 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.871411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.873840 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.873846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.873847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.873853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.875498 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.875887 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.875890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.875890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.875951 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.876155 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.876156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.876156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.876290 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.876447 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.876517 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.876654 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:23:54.880219 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.873500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.19.65.189:43353)
0619 10:23:54.873547 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.873549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.873638 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:23:54.873708 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.873709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.873718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.873769 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.874093 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.874718 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.875702 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.875899 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.875976 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.877875 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.877876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.877877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.877883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.879051 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.879418 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.879419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.879420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.879494 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.879686 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.879687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.879688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.879839 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.879914 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.880091 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.880163 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:23:54.880235 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.873427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:43353 (local address 127.0.0.1:60144)
0619 10:23:54.873573 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.873583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.873650 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.873772 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.873773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.873925 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.873975 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.873977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.874800 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.874801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.875608 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.875610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.875976 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.875977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.876043 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.876457 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.876462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.877786 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.879140 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.879141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.879141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.879342 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.879556 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.879557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.879557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.879609 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.879917 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.879917 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.879983 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.880052 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.880137 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 10:23:54.880666 741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:23:54.885193 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.877471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.19.65.190:46713)
0619 10:23:54.877500 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.877502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.877512 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:54.877572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.877572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.877578 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.877641 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.878053 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.878695 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.879313 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.879530 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.879598 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.882404 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.882405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.882405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.882410 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.883997 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.884316 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.884317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.884317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.884374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.884518 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.884518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.884519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.884648 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.884811 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.885014 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.885083 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:23:54.887544 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.880921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52142)
0619 10:23:54.881006 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.881016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.881067 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.881316 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.881316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.881424 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.881465 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.881467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.882211 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.882212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.882897 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.882899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.883271 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.883272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.883347 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.883713 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.883716 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.884883 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.886654 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.886655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.886655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.886864 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.887036 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.887037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.887037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.887094 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.887326 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.887327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.887368 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.887429 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.887494 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 10:23:54.887631 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.880963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.19.65.188:46153)
0619 10:23:54.881002 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.881003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.881054 (+ 51us) server_negotiation.cc:407] Connection header received
0619 10:23:54.881123 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.881124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.881212 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.881286 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.881523 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.882138 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.883010 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.883204 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.883272 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.884967 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.884968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.884968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.884975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.886585 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.886931 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.886932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.886932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.886977 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.887149 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.887149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.887149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.887232 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.887371 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.887452 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.887551 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":26}
I20260619 10:23:54.888868 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: Bootstrap starting.
I20260619 10:23:54.889706 741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:54.889986 741 log.cc:824] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: Log is configured to *not* fsync() on all Append() calls
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:23:54.890769 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: No bootstrap required, opened a new log
I20260619 10:23:54.892714 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.885772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.19.65.189:43353)
0619 10:23:54.885897 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.885898 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.885904 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:23:54.885920 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.885921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.885928 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.885971 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.886549 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.887226 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.888207 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.888395 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.888468 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.890509 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.890510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.890510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.890516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.891798 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.892155 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.892155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.892156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.892196 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.892346 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.892346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.892346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.892422 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.892537 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.892609 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.892659 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 10:23:54.892666 741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.892807 741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:54.892843 741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2fd9800eb7456687f55e25b15291b9, State: Initialized, Role: FOLLOWER
I20260619 10:23:54.892951 741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.893220 741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:54.893219 747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:54.893329 747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:23:54.894750 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 10:23:54.895107 758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:23:54.895159 758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:23:54.895316 591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } attempt: 2
I20260619 10:23:54.896390 673 master_runner.cc:186] Error getting master registration for 127.19.65.190:46713: OK, instance_id { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" instance_seqno: 1781864634554017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:23:54.899020 673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:23:54.902745 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.895539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52146)
0619 10:23:54.895580 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.895588 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.895626 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.895874 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.895874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.896029 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.896071 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.896072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.896920 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.896921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.897731 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.897733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.898204 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.898205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.898265 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.898672 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.898677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.900067 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.901726 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.901727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.901728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.901940 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.902134 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.902135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.902135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.902189 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.902470 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.902471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.902511 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.902570 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.902652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:23:54.903008 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.895674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.19.65.188:46153)
0619 10:23:54.895721 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.895723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.895747 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:23:54.895765 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.895766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.895774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.895822 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.896171 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.896815 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.897813 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.898104 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.898159 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.900156 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.900157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.900157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.900163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.901638 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.902020 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.902021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.902022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.902070 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.902250 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.902251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.902251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.902382 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.902443 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.902856 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.902926 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:23:54.905294 591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: Bootstrap starting.
I20260619 10:23:54.906037 591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:54.906368 591 log.cc:824] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:54.907207 591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: No bootstrap required, opened a new log
I20260619 10:23:54.909377 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.896588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.19.65.190:46713)
0619 10:23:54.896616 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.896618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.897005 (+ 387us) server_negotiation.cc:407] Connection header received
0619 10:23:54.897088 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.897089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.897097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.897124 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.898784 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.899443 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.902093 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.902370 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.902408 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.904557 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.904558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.904559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.904565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.906120 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.906986 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.906987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.906988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.907042 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.907872 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.907873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.907873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.907993 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.908060 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.909218 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.909286 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:23:54.909422 591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.909590 591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:54.909628 591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bdbff64464b44bfb7d96d7905db9f86, State: Initialized, Role: FOLLOWER
I20260619 10:23:54.909761 591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.910075 591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:54.910650 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.896530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.19.65.189:43353)
0619 10:23:54.896561 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.896563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.897865 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 10:23:54.898007 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.898007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.898015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.898041 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.899585 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.900284 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.902695 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.902992 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.903031 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.907121 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.907123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.907123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.907129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.908694 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.909500 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.909501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.909502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.909563 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.909982 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.909983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.909983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.910098 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.910178 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.910445 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.910529 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:23:54.911520 769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:54.911698 769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:23:54.911993 780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:23:54.912030 780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:23:54.912565 523 master_runner.cc:186] Error getting master registration for 127.19.65.189:43353: OK, instance_id { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" instance_seqno: 1781864634697809 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:23:54.914685 523 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:23:54.918437 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.896840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.19.65.188:46153)
0619 10:23:54.897920 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.897921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.899616 (+ 1695us) server_negotiation.cc:407] Connection header received
0619 10:23:54.899635 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.899636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.899656 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.899708 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.900617 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.901567 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.902438 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.903188 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.903248 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.904496 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.904497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.904497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.904503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.917040 (+ 12537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.917430 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.917431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.917431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.917480 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.917677 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.917678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.917678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.917793 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.918221 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.918260 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.918332 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":61,"threads_started":1}
I20260619 10:23:54.931064 666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } attempt: 3
Jun 19 10:23:54 dist-test-slave-xd2q krb5kdc[510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864634, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:23:54.939368 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.931381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:46153 (local address 127.0.0.1:52156)
0619 10:23:54.931448 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:54.931458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:54.931497 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:54.931631 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:54.931632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:54.931801 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:54.931851 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.931853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.932837 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.932838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.933623 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:54.933625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.934173 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.934174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:54.934231 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.934707 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:54.934712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:54.936258 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:54.938265 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.938266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.938267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.938483 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.938694 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:54.938695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:54.938695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:54.938758 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:54.939036 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:54.939037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:54.939079 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:23:54.939150 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:23:54.939236 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 10:23:54.939375 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:54.931435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.19.65.188:46153)
0619 10:23:54.931475 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:54.931477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:54.931486 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:54.931558 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:54.931559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:54.931569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:54.931628 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:54.931922 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.932710 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.933737 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:54.934090 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:54.934162 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:54.936332 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:54.936359 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:54.936359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:54.936366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:54.938180 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.938559 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.938560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.938560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.938620 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:54.938823 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:54.938823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:54.938824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:54.938967 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:54.939034 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:54.939186 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 10:23:54.939265 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:23:54.941285 666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: Bootstrap starting.
I20260619 10:23:54.941978 666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:54.942266 666 log.cc:824] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:54.943686 666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: No bootstrap required, opened a new log
I20260619 10:23:54.945691 666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.945822 666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:54.945856 666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e35dcad6fb4895ba3129e3a270b295, State: Initialized, Role: FOLLOWER
I20260619 10:23:54.945971 666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:54.946208 783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:54.946257 666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:54.946334 783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:23:54.948158 794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:23:54.948236 794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:23:54.950124 598 master_runner.cc:263] No leader master found from master 93e35dcad6fb4895ba3129e3a270b295
I20260619 10:23:54.950177 598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:23:55.045468 783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:23:55.045600 783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:55.045898 783 leader_election.cc:288] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bdbff64464b44bfb7d96d7905db9f86 (127.19.65.190:46713), 6c2fd9800eb7456687f55e25b15291b9 (127.19.65.188:46153)
I20260619 10:23:55.046200 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e35dcad6fb4895ba3129e3a270b295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bdbff64464b44bfb7d96d7905db9f86" is_pre_election: true
I20260619 10:23:55.046180 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e35dcad6fb4895ba3129e3a270b295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2fd9800eb7456687f55e25b15291b9" is_pre_election: true
I20260619 10:23:55.046342 715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e35dcad6fb4895ba3129e3a270b295 in term 0.
I20260619 10:23:55.046339 565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e35dcad6fb4895ba3129e3a270b295 in term 0.
I20260619 10:23:55.046559 616 leader_election.cc:302] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2fd9800eb7456687f55e25b15291b9, 93e35dcad6fb4895ba3129e3a270b295; no voters:
I20260619 10:23:55.046686 783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:23:55.046737 783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:23:55.046768 783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:55.047737 783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:55.047894 783 leader_election.cc:288] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [CANDIDATE]: Term 1 election: Requested vote from peers 3bdbff64464b44bfb7d96d7905db9f86 (127.19.65.190:46713), 6c2fd9800eb7456687f55e25b15291b9 (127.19.65.188:46153)
I20260619 10:23:55.048157 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e35dcad6fb4895ba3129e3a270b295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bdbff64464b44bfb7d96d7905db9f86"
I20260619 10:23:55.048156 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e35dcad6fb4895ba3129e3a270b295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2fd9800eb7456687f55e25b15291b9"
I20260619 10:23:55.048235 565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:55.048235 715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:55.049098 565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e35dcad6fb4895ba3129e3a270b295 in term 1.
I20260619 10:23:55.049098 715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e35dcad6fb4895ba3129e3a270b295 in term 1.
I20260619 10:23:55.049358 615 leader_election.cc:302] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bdbff64464b44bfb7d96d7905db9f86, 93e35dcad6fb4895ba3129e3a270b295; no voters:
I20260619 10:23:55.049472 783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:55.049669 783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [term 1 LEADER]: Becoming Leader. State: Replica: 93e35dcad6fb4895ba3129e3a270b295, State: Running, Role: LEADER
I20260619 10:23:55.049772 783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } }
I20260619 10:23:55.050155 796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e35dcad6fb4895ba3129e3a270b295. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.050305 796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:55.050549 798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:55.050770 798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:55.052165 715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [term 1 FOLLOWER]: Refusing update from remote peer 93e35dcad6fb4895ba3129e3a270b295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:55.052183 565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [term 1 FOLLOWER]: Refusing update from remote peer 93e35dcad6fb4895ba3129e3a270b295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 10:23:55.052404 544 tablet.cc:2369] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:55.052774 783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:55.052980 796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:55.054028 769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e35dcad6fb4895ba3129e3a270b295. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.054131 769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:23:55.054260 747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e35dcad6fb4895ba3129e3a270b295. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.054358 747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:23:55.055823 796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.055923 796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:55.056413 747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.056499 747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:23:55.056666 783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.056749 783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:55.057021 799 mvcc.cc:205] Tried to move back new op lower bound from 7298517545171394560 to 7298517545164324864. Current Snapshot: MvccSnapshot[applied={T|T < 7298517545171394560}]
I20260619 10:23:55.057040 769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93e35dcad6fb4895ba3129e3a270b295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bdbff64464b44bfb7d96d7905db9f86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46713 } } peers { permanent_uuid: "93e35dcad6fb4895ba3129e3a270b295" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 43353 } } peers { permanent_uuid: "6c2fd9800eb7456687f55e25b15291b9" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 46153 } } }
I20260619 10:23:55.057116 769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:23:55.057117 802 mvcc.cc:205] Tried to move back new op lower bound from 7298517545171394560 to 7298517545164324864. Current Snapshot: MvccSnapshot[applied={T|T < 7298517545171394560 or (T in {7298517545171394560})}]
I20260619 10:23:55.057652 800 mvcc.cc:205] Tried to move back new op lower bound from 7298517545171394560 to 7298517545164324864. Current Snapshot: MvccSnapshot[applied={T|T < 7298517545171394560}]
I20260619 10:23:55.058118 798 catalog_manager.cc:1347] Generated new cluster ID: 183a354f58574e45a2cea973ad53979a
I20260619 10:23:55.058171 798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:23:55.065259 798 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:23:55.065781 798 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:55.071785 798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93e35dcad6fb4895ba3129e3a270b295: Generated new TSK 0
I20260619 10:23:55.071970 798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:23:55.895588 758 catalog_manager.cc:1259] Loaded cluster ID: 183a354f58574e45a2cea973ad53979a
I20260619 10:23:55.895656 758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: loading cluster ID for follower catalog manager: success
I20260619 10:23:55.896515 758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: acquiring CA information for follower catalog manager: success
I20260619 10:23:55.896761 758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c2fd9800eb7456687f55e25b15291b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:23:55.912367 780 catalog_manager.cc:1259] Loaded cluster ID: 183a354f58574e45a2cea973ad53979a
I20260619 10:23:55.912421 780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: loading cluster ID for follower catalog manager: success
I20260619 10:23:55.913206 780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: acquiring CA information for follower catalog manager: success
I20260619 10:23:55.913443 780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bdbff64464b44bfb7d96d7905db9f86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:23:56.135162 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 523
I20260619 10:23:56.140074 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 598
I20260619 10:23:56.144512 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 673
2026-06-19T10:23:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2843 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:23:56 dist-test-slave-xd2q krb5kdc[813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:23:56 dist-test-slave-xd2q krb5kdc[813](info): set up 2 sockets
Jun 19 10:23:56 dist-test-slave-xd2q krb5kdc[813](info): commencing operation
krb5kdc: starting...
W20260619 10:23:57.175396 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:23:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:23:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:23:57.289498 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37183
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:57.368091 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:57.368330 826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:57.368364 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:57.369505 826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:57.369570 826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:57.369596 826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:57.369617 826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:57.369643 826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:57.371081 826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:57.371371 826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:57.371582 826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:57.376183 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:57.376212 832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:57.376235 826 server_base.cc:1034] running on GCE node
W20260619 10:23:57.376188 831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:57.376734 826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:57.377008 826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:57.378142 826 hybrid_clock.cc:648] HybridClock initialized: now 1781864637378125 us; error 34 us; skew 500 ppm
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864637, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:57.380561 826 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:57.380996 826 webserver.cc:460] Webserver started at http://127.19.65.190:36307/ using document root <none> and password file <none>
I20260619 10:23:57.381218 826 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:57.381260 826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:57.381421 826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:57.382405 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "6f935592afd24e9a865e541e3c674ac6"
format_stamp: "Formatted at 2026-06-19 10:23:57 on dist-test-slave-xd2q"
server_key: "d6e1db52063f881d9e8cc6f91cc8f609"
server_key_iv: "aa8805e50f94dcd5e26296f0709799c4"
server_key_version: "encryptionkey@0"
I20260619 10:23:57.382746 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "6f935592afd24e9a865e541e3c674ac6"
format_stamp: "Formatted at 2026-06-19 10:23:57 on dist-test-slave-xd2q"
server_key: "d6e1db52063f881d9e8cc6f91cc8f609"
server_key_iv: "aa8805e50f94dcd5e26296f0709799c4"
server_key_version: "encryptionkey@0"
I20260619 10:23:57.384078 826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:23:57.384750 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:57.384907 826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:57.384989 826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "6f935592afd24e9a865e541e3c674ac6"
format_stamp: "Formatted at 2026-06-19 10:23:57 on dist-test-slave-xd2q"
server_key: "d6e1db52063f881d9e8cc6f91cc8f609"
server_key_iv: "aa8805e50f94dcd5e26296f0709799c4"
server_key_version: "encryptionkey@0"
I20260619 10:23:57.385062 826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:57.397804 826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:57.399930 826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:57.400075 826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:57.403941 826 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:37183
I20260619 10:23:57.403971 893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:37183 every 8 connection(s)
I20260619 10:23:57.404279 826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:57.405058 894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:57.407161 894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6: Bootstrap starting.
I20260619 10:23:57.407727 894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:57.408000 894 log.cc:824] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:57.408645 894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6: No bootstrap required, opened a new log
I20260619 10:23:57.410172 894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } }
I20260619 10:23:57.410296 894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:57.410321 894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f935592afd24e9a865e541e3c674ac6, State: Initialized, Role: FOLLOWER
I20260619 10:23:57.410478 894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } }
I20260619 10:23:57.410537 894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:57.410557 894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:57.410625 894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:57.411115 894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } }
I20260619 10:23:57.411216 894 leader_election.cc:302] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f935592afd24e9a865e541e3c674ac6; no voters:
I20260619 10:23:57.411397 894 leader_election.cc:288] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:23:57.411453 897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:57.411623 897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f935592afd24e9a865e541e3c674ac6, State: Running, Role: LEADER
I20260619 10:23:57.411638 894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:57.411712 897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } }
I20260619 10:23:57.412233 898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f935592afd24e9a865e541e3c674ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } } }
I20260619 10:23:57.412340 898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:57.412640 906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:57.412746 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f935592afd24e9a865e541e3c674ac6. Latest consensus state: current_term: 1 leader_uuid: "6f935592afd24e9a865e541e3c674ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f935592afd24e9a865e541e3c674ac6" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 37183 } } }
I20260619 10:23:57.412839 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6 [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:57.413125 906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:57.413828 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 826
I20260619 10:23:57.413949 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:23:57.414149 19718 external_mini_cluster.cc:1428] Setting key fccbf1782c15a237b4a6ecd336e2dc23
I20260619 10:23:57.414942 906 catalog_manager.cc:1347] Generated new cluster ID: 3abefa67a45940af8f3d58b0770987fa
I20260619 10:23:57.414999 906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:57.425101 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:57.414731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.19.65.190:37183)
0619 10:23:57.414883 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:57.414891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:57.414928 (+ 37us) server_negotiation.cc:407] Connection header received
0619 10:23:57.415043 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:57.415050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:57.415102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:57.415236 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:57.415952 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:57.417376 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:57.418377 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:57.418982 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:57.419086 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:57.421711 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:57.421715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:57.421715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:57.421726 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:57.423571 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:57.423962 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:57.423965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:57.423965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:57.424061 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:57.424254 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:57.424255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:57.424255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:57.424528 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:57.424668 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:57.424763 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:23:57.424889 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260619 10:23:57.425855 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:23:57.432209 906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:23:57.432888 906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:57.449663 906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f935592afd24e9a865e541e3c674ac6: Generated new TSK 0
I20260619 10:23:57.449860 906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:23:57.479668 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 826
2026-06-19T10:23:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1333 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[921](info): set up 2 sockets
Jun 19 10:23:57 dist-test-slave-xd2q krb5kdc[921](info): commencing operation
krb5kdc: starting...
W20260619 10:23:58.499940 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:23:58 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:23:58.607859 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:58.684429 934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:58.684657 934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:58.684693 934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:58.685786 934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:58.685851 934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:58.685874 934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:58.685906 934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:58.685932 934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:58.687367 934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:58.687618 934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:58.687829 934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:58.692559 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:58.692567 939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:58.692571 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:58.692595 934 server_base.cc:1034] running on GCE node
I20260619 10:23:58.693002 934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:58.693305 934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:58.694459 934 hybrid_clock.cc:648] HybridClock initialized: now 1781864638694451 us; error 33 us; skew 500 ppm
Jun 19 10:23:58 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864638, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:58.696957 934 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:58.697309 934 webserver.cc:460] Webserver started at http://127.19.65.190:37999/ using document root <none> and password file <none>
I20260619 10:23:58.697487 934 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:58.697525 934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:58.697611 934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:58.698549 934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "ae1856a171da99a3f758a0fac8820635"
server_key_iv: "a552ed660e4e7dd6be6407f988048403"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.698832 934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "ae1856a171da99a3f758a0fac8820635"
server_key_iv: "a552ed660e4e7dd6be6407f988048403"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.700111 934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:23:58.700734 949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:58.700897 934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:23:58.700974 934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "ae1856a171da99a3f758a0fac8820635"
server_key_iv: "a552ed660e4e7dd6be6407f988048403"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.701025 934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:58.726967 934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:58.728796 934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:58.728893 934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:58.732681 934 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36015
I20260619 10:23:58.732754 1001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36015 every 8 connection(s)
I20260619 10:23:58.733039 934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:58.733950 1002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:58.736083 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap starting.
I20260619 10:23:58.736570 1002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:58.736793 1002 log.cc:824] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:58.737398 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: No bootstrap required, opened a new log
I20260619 10:23:58.738878 1002 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:58.738999 1002 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:58.739020 1002 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Initialized, Role: FOLLOWER
I20260619 10:23:58.739141 1002 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:58.739215 1002 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:58.739238 1002 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:58.739272 1002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:58.740046 1002 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:58.740126 1002 leader_election.cc:302] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45bdd17bea14c8ba8edc23aa88f2d5e; no voters:
I20260619 10:23:58.740253 1002 leader_election.cc:288] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:23:58.740319 1005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:58.740450 1005 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 LEADER]: Becoming Leader. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Running, Role: LEADER
I20260619 10:23:58.740504 1002 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:58.740556 1005 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:58.740916 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45bdd17bea14c8ba8edc23aa88f2d5e. Latest consensus state: current_term: 1 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:23:58.740895 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:23:58.741043 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:58.741035 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:58.741484 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:58.741844 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 934
I20260619 10:23:58.741950 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:23:58.742046 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:58.742152 19718 external_mini_cluster.cc:1428] Setting key 84327c8b5bf0b389dd728ad0e2a82c1f
I20260619 10:23:58.743983 1013 catalog_manager.cc:1347] Generated new cluster ID: 803a7867884a4c5283b95b167b83a921
I20260619 10:23:58.744047 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:23:58 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:58.761157 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:58.742584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.19.65.190:36015)
0619 10:23:58.742755 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:58.742764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:58.742798 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:23:58.742995 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:58.743003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:58.743078 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:58.743226 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:58.751422 (+ 8196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.752720 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.753769 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.754350 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.754439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:58.757378 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:58.757382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:58.757382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:58.757395 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:58.759479 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:58.759878 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:58.759881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:58.759882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:58.759977 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:58.760150 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:58.760151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:58.760151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:58.760409 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:58.760558 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:58.760813 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 10:23:58.760952 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260619 10:23:58.764640 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:23:58.765082 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:58.768889 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Generated new TSK 0
I20260619 10:23:58.769037 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:23:58.795046 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:58.876179 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:58.876348 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:58.876370 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:58.877593 1028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:58.877648 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:58.877674 1028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:23:58.879135 1028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:58.879359 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:58.879588 1028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:58.884227 1033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:58.884227 1034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:58.884284 1028 server_base.cc:1034] running on GCE node
W20260619 10:23:58.884266 1036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:58.884624 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:58.884894 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:58.886044 1028 hybrid_clock.cc:648] HybridClock initialized: now 1781864638886040 us; error 30 us; skew 500 ppm
Jun 19 10:23:58 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864638, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:58.888516 1028 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:23:58.888998 1028 webserver.cc:460] Webserver started at http://127.19.65.129:42945/ using document root <none> and password file <none>
I20260619 10:23:58.889218 1028 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:58.889303 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:58.889427 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:58.890540 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "7877ae4a97e34df3b9df88be5c178d87"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "2247090af2c6683886d612e3d2027e01"
server_key_iv: "fbcd7e3c736ea1d91246ee65ba91dfdf"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.890902 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "7877ae4a97e34df3b9df88be5c178d87"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "2247090af2c6683886d612e3d2027e01"
server_key_iv: "fbcd7e3c736ea1d91246ee65ba91dfdf"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.892206 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:23:58.892845 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:58.892999 1028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:23:58.893086 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "7877ae4a97e34df3b9df88be5c178d87"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "2247090af2c6683886d612e3d2027e01"
server_key_iv: "fbcd7e3c736ea1d91246ee65ba91dfdf"
server_key_version: "encryptionkey@0"
I20260619 10:23:58.893157 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:58.906674 1028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:58.908527 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:58.908675 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:58.908891 1028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:58.909196 1028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:58.909245 1028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:58.909277 1028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:58.909307 1028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:58.914880 1028 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40905
I20260619 10:23:58.914932 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40905 every 8 connection(s)
I20260619 10:23:58.915215 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:23:58.919219 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1028
I20260619 10:23:58.919353 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:23:58.919545 19718 external_mini_cluster.cc:1428] Setting key 086d2320d8ec4212acfc38c9f828542b
Jun 19 10:23:58 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864638, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:58.925563 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:58.915875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:39221 (local address 127.19.65.190:36015)
0619 10:23:58.915938 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:58.915941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:58.916139 (+ 198us) server_negotiation.cc:407] Connection header received
0619 10:23:58.916599 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:58.916600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:58.916610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:58.916676 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:58.917552 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.918122 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.919095 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.919362 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.919441 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:58.922156 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:58.922158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:58.922158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:58.922166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:58.923891 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:58.924422 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:58.924424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:58.924424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:58.924491 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:58.924808 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:58.924810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:58.924810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:58.924957 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:58.925040 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:58.925390 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 10:23:58.925476 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:23:58.925750 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:58.915853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:39221)
0619 10:23:58.916030 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:58.916063 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:58.916522 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:58.916727 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:58.916730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:58.917272 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:58.917450 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:58.917454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.918213 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.918214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:58.918989 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:58.918991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:58.919462 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:58.919463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:58.919585 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:58.920127 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:58.920136 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:58.922045 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:58.923965 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:58.923973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:58.923976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:58.924336 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:58.924573 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:58.924574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:58.924574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:58.924724 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:58.925044 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:58.925051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:58.925235 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 10:23:58.925356 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 10:23:58.925509 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260619 10:23:58.926539 1157 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:23:58.926641 1157 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:58.926807 1157 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
I20260619 10:23:58.927416 966 ts_manager.cc:194] Registered new tserver with Master: 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
I20260619 10:23:58.928395 966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:39221
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:23:58.943542 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:59.025846 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:59.026028 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:59.026050 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:59.027143 1162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:59.027177 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:59.027202 1162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:23:59.028568 1162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:59.028729 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:59.028894 1162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:59.033506 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.033532 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.033524 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.033723 1162 server_base.cc:1034] running on GCE node
I20260619 10:23:59.033963 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:59.034209 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:59.035343 1162 hybrid_clock.cc:648] HybridClock initialized: now 1781864639035335 us; error 36 us; skew 500 ppm
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:59.037878 1162 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:23:59.038224 1162 webserver.cc:460] Webserver started at http://127.19.65.130:38443/ using document root <none> and password file <none>
I20260619 10:23:59.038390 1162 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:59.038429 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:59.038511 1162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:59.039497 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "e53f940be9494c068a6e8611aa0a7e25"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "d48232513d91c5d03d6769f9221db51f"
server_key_iv: "8ac5229775d687da450bbc2c64eb0027"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.039785 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "e53f940be9494c068a6e8611aa0a7e25"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "d48232513d91c5d03d6769f9221db51f"
server_key_iv: "8ac5229775d687da450bbc2c64eb0027"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.041133 1162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:23:59.041939 1177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.042095 1162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:59.042207 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "e53f940be9494c068a6e8611aa0a7e25"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "d48232513d91c5d03d6769f9221db51f"
server_key_iv: "8ac5229775d687da450bbc2c64eb0027"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.042305 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:59.056622 1162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:59.058318 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:59.058418 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:59.058584 1162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:59.058811 1162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:59.058840 1162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.058859 1162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:59.058872 1162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.064441 1162 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34353
I20260619 10:23:59.064488 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:34353 every 8 connection(s)
I20260619 10:23:59.064759 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:23:59.067575 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1162
I20260619 10:23:59.067687 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:23:59.067855 19718 external_mini_cluster.cc:1428] Setting key fea8187b17bbeffa174d43d308379f35
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:59.074241 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.065468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:52431 (local address 127.19.65.190:36015)
0619 10:23:59.065524 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.065537 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.065706 (+ 169us) server_negotiation.cc:407] Connection header received
0619 10:23:59.066228 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.066230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.066242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.066288 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:59.067186 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.067981 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.069104 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.069312 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.069383 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.071565 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:59.071566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:59.071566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:59.071573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:59.072922 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.073310 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.073311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.073311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.073365 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.073596 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.073598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.073598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.073705 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:59.073774 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.074067 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.074152 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:23:59.074426 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.065447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:52431)
0619 10:23:59.065593 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.065630 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.066125 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.066362 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.066366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.066914 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:59.067087 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.067092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.068062 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.068063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.069019 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.069022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.069387 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.069388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.069511 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.069989 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:59.069999 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:59.071482 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:59.073016 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.073023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.073025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.073231 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.073430 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.073431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.073431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.073535 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.073765 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:59.073770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:59.073906 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.074043 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.074187 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260619 10:23:59.074801 1291 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:23:59.074885 1291 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:59.075052 1291 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:23:59.075434 966 ts_manager.cc:194] Registered new tserver with Master: e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353)
I20260619 10:23:59.075799 966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:52431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:23:59.091013 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:59.170748 1296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:59.170967 1296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:59.171003 1296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:59.172139 1296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:59.172199 1296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:59.172242 1296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:23:59.173666 1296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:59.173923 1296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:59.174185 1296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:59.179039 1304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.179064 1302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.179111 1296 server_base.cc:1034] running on GCE node
W20260619 10:23:59.179062 1301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.179441 1296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:59.179672 1296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:59.180825 1296 hybrid_clock.cc:648] HybridClock initialized: now 1781864639180812 us; error 25 us; skew 500 ppm
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:59.183198 1296 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:23:59.183610 1296 webserver.cc:460] Webserver started at http://127.19.65.131:35401/ using document root <none> and password file <none>
I20260619 10:23:59.183794 1296 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:59.183833 1296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:59.183923 1296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:23:59.184939 1296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "17150e9a2d19e338aedd38fac7d34241"
server_key_iv: "3e12830f749a0504e6e6308e36d6a1a8"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.185252 1296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "17150e9a2d19e338aedd38fac7d34241"
server_key_iv: "3e12830f749a0504e6e6308e36d6a1a8"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.186501 1296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:23:59.187150 1311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.187361 1296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.187465 1296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "17150e9a2d19e338aedd38fac7d34241"
server_key_iv: "3e12830f749a0504e6e6308e36d6a1a8"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.187538 1296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:59.208741 1296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:59.210528 1296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:59.210672 1296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:59.210904 1296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:59.211236 1296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:23:59.211269 1296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.211338 1296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:23:59.211367 1296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.216867 1296 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41813
I20260619 10:23:59.216943 1424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:41813 every 8 connection(s)
I20260619 10:23:59.217260 1296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:59.225210 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1296
I20260619 10:23:59.225281 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:23:59.225476 19718 external_mini_cluster.cc:1428] Setting key 3d3f24b00733c91284f712d0edf9686b
I20260619 10:23:59.226606 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.217973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:39727 (local address 127.19.65.190:36015)
0619 10:23:59.218017 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.218019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.218192 (+ 173us) server_negotiation.cc:407] Connection header received
0619 10:23:59.218759 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.218761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.218773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.218835 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:59.219791 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.220294 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.221287 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.221477 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.221543 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.223699 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:59.223701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:59.223701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:59.223710 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:59.225081 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.225566 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.225567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.225568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.225624 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.225959 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.225960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.225960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.226115 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:59.226177 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.226474 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.226545 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:23:59.226786 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.217932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:39727)
0619 10:23:59.218075 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.218117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.218659 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.218902 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.218906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.219518 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:59.219699 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.219704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.220404 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.220405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.221189 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.221192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.221561 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.221562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.221679 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.222174 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:59.222182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:59.223606 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:59.225176 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.225182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.225184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.225471 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.225709 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.225710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.225710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.225864 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.226178 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:59.226184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:59.226333 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.226442 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.226588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260619 10:23:59.227218 1425 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:23:59.227344 1425 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:59.227526 1425 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
I20260619 10:23:59.227895 966 ts_manager.cc:194] Registered new tserver with Master: 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
I20260619 10:23:59.228418 966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:39727
I20260619 10:23:59.237130 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:23:59.245507 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.238864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.19.65.190:36015)
0619 10:23:59.238929 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.238931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.239052 (+ 121us) server_negotiation.cc:407] Connection header received
0619 10:23:59.239128 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.239130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.239140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.239183 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:59.239822 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.240370 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.241342 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.241547 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.241657 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.242897 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:59.242898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:59.242898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:59.242906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:59.244297 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.244714 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.244715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.244716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.244771 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.244972 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.244973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.244974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.245119 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:59.245219 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.245322 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.245407 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:23:59.247249 966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:23:59.250185 966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:23:59.257651 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.0.0.1:54430)
0619 10:23:59.252996 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.253006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.253060 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.253690 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.253690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.253692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.253736 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.253738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.255005 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.255006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.256396 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.256398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257340 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.257441 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.257461 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.257510 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.257570 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":73,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1}
I20260619 10:23:59.257982 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:34353 (local address 127.0.0.1:43566)
0619 10:23:59.252890 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.252935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.253112 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.253559 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.253560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.253562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.253616 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.253618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.255104 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.255106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.256522 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.256525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257701 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.257782 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.257787 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.257836 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.257885 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260619 10:23:59.258085 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.19.65.129:40905)
0619 10:23:59.253526 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.253530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.253540 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:59.253576 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.253577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.253593 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.253805 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.253882 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.254897 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.256513 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257241 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257868 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.257933 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.257957 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.257999 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":111,"threads_started":1}
I20260619 10:23:59.258322 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.19.65.130:34353)
0619 10:23:59.252953 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.252959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.253020 (+ 61us) server_negotiation.cc:407] Connection header received
0619 10:23:59.253362 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.253365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.253383 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.253473 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.253694 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.254979 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.256634 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257510 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257588 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.257650 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.258160 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.258213 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260619 10:23:59.258734 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.19.65.131:41813)
0619 10:23:59.253242 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.253246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.253256 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:59.253290 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.253292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.253308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.253374 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.253565 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.254695 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.256515 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257550 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257590 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.257648 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.258604 (+ 956us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.258656 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":74,"threads_started":1}
I20260619 10:23:59.258775 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3d455f569c45b9b5220558c1545573 (DEFAULT_TABLE table=test-table [id=05a954bc68a245c2ba8782d1065304dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:59.259127 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.252602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41813 (local address 127.0.0.1:52852)
0619 10:23:59.253158 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.253167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.253205 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.253389 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.253392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.253395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.253461 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.253463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.254813 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.254814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.256388 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.256391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.257978 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.257978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.258281 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.258286 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.259001 (+ 715us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.259048 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":71,"thread_start_us":66,"threads_started":1}
I20260619 10:23:59.259017 1225 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3d455f569c45b9b5220558c1545573 (DEFAULT_TABLE table=test-table [id=05a954bc68a245c2ba8782d1065304dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:59.259145 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3d455f569c45b9b5220558c1545573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:59.259385 1225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3d455f569c45b9b5220558c1545573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:59.262015 1439 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap starting.
I20260619 10:23:59.262753 1439 tablet_bootstrap.cc:654] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:59.262791 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3d455f569c45b9b5220558c1545573 (DEFAULT_TABLE table=test-table [id=05a954bc68a245c2ba8782d1065304dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:23:59.263135 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3d455f569c45b9b5220558c1545573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:23:59.263643 1439 log.cc:824] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:59.264544 1439 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: No bootstrap required, opened a new log
I20260619 10:23:59.264618 1439 ts_tablet_manager.cc:1397] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 10:23:59.266459 1442 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Bootstrap starting.
I20260619 10:23:59.266731 1440 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Bootstrap starting.
I20260619 10:23:59.266716 1439 raft_consensus.cc:348] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.266901 1439 raft_consensus.cc:374] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:59.266945 1439 raft_consensus.cc:729] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877ae4a97e34df3b9df88be5c178d87, State: Initialized, Role: FOLLOWER
I20260619 10:23:59.267108 1439 consensus_queue.cc:260] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.267411 1439 ts_tablet_manager.cc:1428] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:23:59.267432 1440 tablet_bootstrap.cc:654] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:59.267544 1157 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
I20260619 10:23:59.267928 1442 tablet_bootstrap.cc:654] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Neither blocks nor log segments found. Creating new log.
I20260619 10:23:59.268023 1440 log.cc:824] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:59.268644 1442 log.cc:824] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:59.268787 1440 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: No bootstrap required, opened a new log
I20260619 10:23:59.268860 1440 ts_tablet_manager.cc:1397] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:23:59.270228 1440 raft_consensus.cc:348] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.270349 1440 raft_consensus.cc:374] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:59.270370 1440 raft_consensus.cc:729] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53f940be9494c068a6e8611aa0a7e25, State: Initialized, Role: FOLLOWER
I20260619 10:23:59.270521 1440 consensus_queue.cc:260] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.270762 1440 ts_tablet_manager.cc:1428] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:23:59.270900 1291 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
I20260619 10:23:59.271461 1442 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: No bootstrap required, opened a new log
I20260619 10:23:59.271538 1442 ts_tablet_manager.cc:1397] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 10:23:59.274080 1442 raft_consensus.cc:348] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.274281 1442 raft_consensus.cc:374] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:23:59.274319 1442 raft_consensus.cc:729] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fafdc5a9514421b3af4c5b391d0ca9, State: Initialized, Role: FOLLOWER
I20260619 10:23:59.274457 1442 consensus_queue.cc:260] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.274746 1442 ts_tablet_manager.cc:1428] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:23:59.274751 1425 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
W20260619 10:23:59.315248 1292 tablet.cc:2369] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:23:59.415992 1158 tablet.cc:2369] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:23:59.467854 1426 tablet.cc:2369] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:23:59.535356 1447 raft_consensus.cc:482] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:23:59.535473 1447 raft_consensus.cc:504] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.535962 1447 leader_election.cc:288] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
I20260619 10:23:59.539840 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.536179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59895 (local address 127.19.65.130:34353)
0619 10:23:59.536246 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.536248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.536258 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:23:59.536356 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.536358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.536370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.536449 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.536653 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.537401 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.538872 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.539485 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.539550 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.539558 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.539733 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.539771 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:23:59.539840 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.536228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38729 (local address 127.19.65.129:40905)
0619 10:23:59.536325 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.536327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.536359 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:23:59.536424 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.536425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.536438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.536491 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.536652 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.537416 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.538867 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.539471 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.539550 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.539558 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.539733 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.539770 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 10:23:59.539866 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.536103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:34353 (local address 127.19.65.131:59895)
0619 10:23:59.536208 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.536220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.536264 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.536484 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.536484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.536486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.536572 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.536574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.537483 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.537484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.538788 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.538791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.539562 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.539563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.539650 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.539652 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.539720 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.539769 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":68}
I20260619 10:23:59.539881 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.536112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.19.65.131:38729)
0619 10:23:59.536307 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.536317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.536370 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.536500 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.536501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.536502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.536572 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.536574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.537491 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.537492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.538791 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.538792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.539581 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.539582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.539646 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.539650 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.539720 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.539769 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":84,"thread_start_us":111,"threads_started":1}
I20260619 10:23:59.540333 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3d455f569c45b9b5220558c1545573" candidate_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877ae4a97e34df3b9df88be5c178d87" is_pre_election: true
I20260619 10:23:59.540290 1245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3d455f569c45b9b5220558c1545573" candidate_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53f940be9494c068a6e8611aa0a7e25" is_pre_election: true
I20260619 10:23:59.540482 1111 raft_consensus.cc:2454] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fafdc5a9514421b3af4c5b391d0ca9 in term 0.
I20260619 10:23:59.540495 1245 raft_consensus.cc:2454] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fafdc5a9514421b3af4c5b391d0ca9 in term 0.
I20260619 10:23:59.540693 1314 leader_election.cc:302] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fafdc5a9514421b3af4c5b391d0ca9, 7877ae4a97e34df3b9df88be5c178d87; no voters:
I20260619 10:23:59.540835 1447 raft_consensus.cc:2781] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:23:59.540903 1447 raft_consensus.cc:482] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:23:59.540920 1447 raft_consensus.cc:3037] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:59.541769 1447 raft_consensus.cc:504] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.541913 1447 leader_election.cc:288] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [CANDIDATE]: Term 1 election: Requested vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
I20260619 10:23:59.542093 1245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3d455f569c45b9b5220558c1545573" candidate_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53f940be9494c068a6e8611aa0a7e25"
I20260619 10:23:59.542156 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3d455f569c45b9b5220558c1545573" candidate_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877ae4a97e34df3b9df88be5c178d87"
I20260619 10:23:59.542181 1245 raft_consensus.cc:3037] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:59.542235 1111 raft_consensus.cc:3037] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:23:59.543098 1111 raft_consensus.cc:2454] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fafdc5a9514421b3af4c5b391d0ca9 in term 1.
I20260619 10:23:59.543098 1245 raft_consensus.cc:2454] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fafdc5a9514421b3af4c5b391d0ca9 in term 1.
I20260619 10:23:59.543280 1312 leader_election.cc:302] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25; no voters:
I20260619 10:23:59.543404 1447 raft_consensus.cc:2781] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:23:59.543582 1447 raft_consensus.cc:686] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 20fafdc5a9514421b3af4c5b391d0ca9, State: Running, Role: LEADER
I20260619 10:23:59.543677 1447 consensus_queue.cc:237] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.544441 966 catalog_manager.cc:5495] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131). New cstate: current_term: 1 leader_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } health_report { overall_health: HEALTHY } } }
I20260619 10:23:59.607286 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.604961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.19.65.131:41813)
0619 10:23:59.605014 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.605016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.605025 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:23:59.605117 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.605118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.605137 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.605216 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:23:59.605350 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.605796 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.606533 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.606742 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.606813 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.606975 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:23:59.607076 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:23:59.607161 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.607186 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.607203 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:23:59.608996 1245 raft_consensus.cc:1261] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Refusing update from remote peer 20fafdc5a9514421b3af4c5b391d0ca9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:59.609011 1111 raft_consensus.cc:1261] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Refusing update from remote peer 20fafdc5a9514421b3af4c5b391d0ca9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:23:59.609347 1447 consensus_queue.cc:1035] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:59.609483 1449 consensus_queue.cc:1035] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:23:59.611703 1452 mvcc.cc:205] Tried to move back new op lower bound from 7298517563836002304 to 7298517563571474432. Current Snapshot: MvccSnapshot[applied={T|T < 7298517563836002304}]
I20260619 10:23:59.611996 1454 mvcc.cc:205] Tried to move back new op lower bound from 7298517563836002304 to 7298517563571474432. Current Snapshot: MvccSnapshot[applied={T|T < 7298517563836002304}]
I20260619 10:23:59.612721 1453 mvcc.cc:205] Tried to move back new op lower bound from 7298517563836002304 to 7298517563571474432. Current Snapshot: MvccSnapshot[applied={T|T < 7298517563836002304}]
I20260619 10:23:59.614009 966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:59.614104 966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:23:59.614820 966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Sending DeleteTablet for 3 replicas of tablet 7f3d455f569c45b9b5220558c1545573
I20260619 10:23:59.615150 1225 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3d455f569c45b9b5220558c1545573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:59 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43566
I20260619 10:23:59.615186 1359 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3d455f569c45b9b5220558c1545573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:59 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:52852
I20260619 10:23:59.615396 1461 tablet_replica.cc:330] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: stopping tablet replica
I20260619 10:23:59.615394 1091 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3d455f569c45b9b5220558c1545573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:59 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:54430
I20260619 10:23:59.615502 1461 raft_consensus.cc:2229] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:23:59.615597 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1028
I20260619 10:23:59.615603 1462 tablet_replica.cc:330] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: stopping tablet replica
I20260619 10:23:59.615610 1461 raft_consensus.cc:2258] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:59.615710 1462 raft_consensus.cc:2229] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:23:59.615810 1462 raft_consensus.cc:2258] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:59.616549 1461 ts_tablet_manager.cc:1905] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:23:59.618462 1461 ts_tablet_manager.cc:1918] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:23:59.618541 1461 log.cc:1196] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/7f3d455f569c45b9b5220558c1545573
I20260619 10:23:59.618772 1461 ts_tablet_manager.cc:1939] T 7f3d455f569c45b9b5220558c1545573 P e53f940be9494c068a6e8611aa0a7e25: Deleting consensus metadata
I20260619 10:23:59.618943 1463 tablet_replica.cc:330] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: stopping tablet replica
I20260619 10:23:59.619064 1463 raft_consensus.cc:2229] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:23:59.619186 1463 raft_consensus.cc:2258] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:59.620993 950 catalog_manager.cc:4841] TS e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): tablet 7f3d455f569c45b9b5220558c1545573 (table test-table [id=05a954bc68a245c2ba8782d1065304dc]) successfully deleted
W20260619 10:23:59.622830 952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40905 (error 108)
I20260619 10:23:59.623456 1463 ts_tablet_manager.cc:1905] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:23:59.623540 1435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:23:59.623105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.0.0.1:50324)
0619 10:23:59.623174 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.623185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.623238 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.623494 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 10:23:59.623653 952 catalog_manager.cc:4584] TS 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f3d455f569c45b9b5220558c1545573: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:23:59.624038 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1162
I20260619 10:23:59.625322 1463 ts_tablet_manager.cc:1918] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:23:59.625397 1463 log.cc:1196] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/7f3d455f569c45b9b5220558c1545573
I20260619 10:23:59.625638 1463 ts_tablet_manager.cc:1939] T 7f3d455f569c45b9b5220558c1545573 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting consensus metadata
I20260619 10:23:59.626025 952 catalog_manager.cc:4841] TS 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): tablet 7f3d455f569c45b9b5220558c1545573 (table test-table [id=05a954bc68a245c2ba8782d1065304dc]) successfully deleted
I20260619 10:23:59.629781 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1296
I20260619 10:23:59.634043 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 934
I20260619 10:23:59.638754 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--webserver_interface=127.19.65.190
--webserver_port=37999
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:23:59.717698 1465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:59.717900 1465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:59.717921 1465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:59.719035 1465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:23:59.719072 1465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:59.719099 1465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:59.719131 1465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:23:59.719161 1465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:23:59.720582 1465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:59.720817 1465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:59.721026 1465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:59.729455 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.729467 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.729626 1465 server_base.cc:1034] running on GCE node
W20260619 10:23:59.729435 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.730042 1465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:59.730320 1465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:59.731452 1465 hybrid_clock.cc:648] HybridClock initialized: now 1781864639731439 us; error 28 us; skew 500 ppm
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:59.736533 1465 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:23:59.736972 1465 webserver.cc:460] Webserver started at http://127.19.65.190:37999/ using document root <none> and password file <none>
I20260619 10:23:59.737217 1465 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:59.737290 1465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:59.738665 1465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:59.739346 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.739562 1465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:23:59.739658 1465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "ae1856a171da99a3f758a0fac8820635"
server_key_iv: "a552ed660e4e7dd6be6407f988048403"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.740062 1465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:59.752612 1465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:59.754338 1465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:59.754485 1465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:59.758394 1465 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36015
I20260619 10:23:59.758463 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36015 every 8 connection(s)
I20260619 10:23:59.758836 1465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:23:59.760408 1534 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:23:59.761080 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap starting.
I20260619 10:23:59.763233 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1465
I20260619 10:23:59.763536 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:40905
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=42945
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:23:59.769171 1534 log.cc:824] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:59.770941 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:23:59.771220 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap complete.
I20260619 10:23:59.773463 1534 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:59.773690 1534 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Initialized, Role: FOLLOWER
I20260619 10:23:59.773872 1534 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:59.773947 1534 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:23:59.774010 1534 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:23:59.774071 1534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:23:59.775009 1534 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:59.775122 1534 leader_election.cc:302] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45bdd17bea14c8ba8edc23aa88f2d5e; no voters:
I20260619 10:23:59.775311 1534 leader_election.cc:288] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:23:59.775347 1538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:23:59.775462 1538 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 LEADER]: Becoming Leader. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Running, Role: LEADER
I20260619 10:23:59.775542 1538 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:23:59.775566 1534 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:23:59.775791 1539 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:23:59.775879 1539 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:59.775789 1538 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45bdd17bea14c8ba8edc23aa88f2d5e. Latest consensus state: current_term: 2 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:23:59.776067 1538 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:23:59.776211 1542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:23:59.777042 1542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:23:59.777159 1542 catalog_manager.cc:1259] Loaded cluster ID: 803a7867884a4c5283b95b167b83a921
I20260619 10:23:59.777438 1542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:23:59.777671 1542 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 10:23:59.780508 1542 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 10:23:59.782202 1542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:23:59.782387 1542 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Loaded TSK: 0
I20260619 10:23:59.782598 1542 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:23:59.874869 1536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:23:59.875130 1536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:23:59.875180 1536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:23:59.876755 1536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:23:59.876813 1536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:23:59.876854 1536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:23:59.878935 1536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:40905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=42945
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:23:59.879169 1536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:23:59.879448 1536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:23:59.884537 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.884541 1561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:23:59.884608 1564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:23:59.884867 1536 server_base.cc:1034] running on GCE node
I20260619 10:23:59.885054 1536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:23:59.885346 1536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:23:59.886526 1536 hybrid_clock.cc:648] HybridClock initialized: now 1781864639886499 us; error 36 us; skew 500 ppm
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:23:59.889546 1536 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:23:59.890062 1536 webserver.cc:460] Webserver started at http://127.19.65.129:42945/ using document root <none> and password file <none>
I20260619 10:23:59.890324 1536 fs_manager.cc:359] Metadata directory not provided
I20260619 10:23:59.890407 1536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:23:59.892094 1536 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:23:59.892880 1571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:23:59.893086 1536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:23:59.893179 1536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "7877ae4a97e34df3b9df88be5c178d87"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "2247090af2c6683886d612e3d2027e01"
server_key_iv: "fbcd7e3c736ea1d91246ee65ba91dfdf"
server_key_version: "encryptionkey@0"
I20260619 10:23:59.893569 1536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:23:59.907405 1536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:23:59.909602 1536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:23:59.909746 1536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:23:59.909972 1536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:23:59.910409 1578 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:23:59.911809 1536 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:23:59.911880 1536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:23:59.911919 1536 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:23:59.912745 1536 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:23:59.912804 1536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:23:59.912866 1578 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap starting.
I20260619 10:23:59.920202 1536 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40905
I20260619 10:23:59.920434 1685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40905 every 8 connection(s)
I20260619 10:23:59.920636 1536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:23:59.922644 1578 log.cc:824] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Log is configured to *not* fsync() on all Append() calls
I20260619 10:23:59.925165 1578 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:23:59.925468 1578 tablet_bootstrap.cc:492] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap complete.
I20260619 10:23:59.926637 1578 ts_tablet_manager.cc:1397] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260619 10:23:59.927855 1578 raft_consensus.cc:348] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.928192 1578 raft_consensus.cc:729] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877ae4a97e34df3b9df88be5c178d87, State: Initialized, Role: FOLLOWER
I20260619 10:23:59.928319 1578 consensus_queue.cc:260] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } }
I20260619 10:23:59.928562 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1536
I20260619 10:23:59.928588 1578 ts_tablet_manager.cc:1428] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260619 10:23:59.928829 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:34353
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=38443
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:23:59 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864639, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:23:59.932998 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.921301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:39015)
0619 10:23:59.921615 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.921657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.922205 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.922919 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.922922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.923158 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:23:59.923403 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.923409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.924978 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.924979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.926386 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.926389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.927200 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.927201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.927368 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.927963 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:23:59.927972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:23:59.929513 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:23:59.931531 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.931537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.931539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.931747 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.931988 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:23:59.931989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:23:59.931989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:23:59.932102 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:23:59.932437 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:23:59.932443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:23:59.932569 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.932667 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.932822 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260619 10:23:59.933009 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.921403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:39015 (local address 127.19.65.190:36015)
0619 10:23:59.922425 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.922433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.922471 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:23:59.922587 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.922593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.922643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.922781 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:23:59.923530 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.924860 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.926485 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.927089 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.927200 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.929626 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:23:59.929630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:23:59.929630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:23:59.929640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:23:59.931446 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.931833 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.931836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.931837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.931922 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:23:59.932164 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:23:59.932165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:23:59.932166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:23:59.932364 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:23:59.932458 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.932718 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.932832 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":83,"threads_started":1}
I20260619 10:23:59.933903 1686 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:23:59.933996 1686 heartbeater.cc:464] Registering TS with master...
I20260619 10:23:59.934182 1686 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
I20260619 10:23:59.934855 1498 ts_manager.cc:194] Registered new tserver with Master: 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
I20260619 10:23:59.935923 1498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:39015
I20260619 10:23:59.936663 1686 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
I20260619 10:23:59.942400 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.935622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.19.65.129:40905)
0619 10:23:59.939559 (+ 3937us) server_negotiation.cc:206] Beginning negotiation
0619 10:23:59.939562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:23:59.939569 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:23:59.939592 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:23:59.939594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:23:59.939604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:23:59.939677 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:23:59.939841 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.940470 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.941463 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.941921 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.942010 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.942066 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:23:59.942293 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 10:23:59.942332 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3913,"thread_start_us":63,"threads_started":1}
I20260619 10:23:59.942405 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:23:59.935411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.0.0.1:50338)
0619 10:23:59.939389 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0619 10:23:59.939417 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 10:23:59.939467 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:23:59.939694 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:23:59.939696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:23:59.939698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:23:59.939768 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.939769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.940575 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.940576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.941377 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:23:59.941379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:23:59.942031 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:23:59.942033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:23:59.942158 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:23:59.942173 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 10:23:59.942276 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:23:59.942329 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3948,"thread_start_us":75,"threads_started":1}
I20260619 10:23:59.942953 1620 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3d455f569c45b9b5220558c1545573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:23:59 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:50338
I20260619 10:23:59.947391 1695 tablet_replica.cc:330] stopping tablet replica
I20260619 10:23:59.947497 1695 raft_consensus.cc:2229] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:23:59.947578 1695 raft_consensus.cc:2258] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:23:59.947850 1695 ts_tablet_manager.cc:1905] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:23:59.949757 1695 ts_tablet_manager.cc:1918] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:23:59.949828 1695 log.cc:1196] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/7f3d455f569c45b9b5220558c1545573
I20260619 10:23:59.950052 1695 ts_tablet_manager.cc:1939] T 7f3d455f569c45b9b5220558c1545573 P 7877ae4a97e34df3b9df88be5c178d87: Deleting consensus metadata
I20260619 10:23:59.950490 1484 catalog_manager.cc:4841] TS 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905): tablet 7f3d455f569c45b9b5220558c1545573 (table test-table [id=05a954bc68a245c2ba8782d1065304dc]) successfully deleted
W20260619 10:24:00.026516 1692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.026719 1692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.026742 1692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.027961 1692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.028003 1692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.028029 1692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:24:00.029458 1692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:34353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=38443
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.029702 1692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.029932 1692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:00.034838 1700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.034885 1692 server_base.cc:1034] running on GCE node
W20260619 10:24:00.034843 1703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.034844 1701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.035226 1692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.035506 1692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.036667 1692 hybrid_clock.cc:648] HybridClock initialized: now 1781864640036657 us; error 35 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.039418 1692 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:24:00.039800 1692 webserver.cc:460] Webserver started at http://127.19.65.130:38443/ using document root <none> and password file <none>
I20260619 10:24:00.039975 1692 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.040026 1692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.041260 1692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:00.041765 1710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.041905 1692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.041994 1692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "e53f940be9494c068a6e8611aa0a7e25"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "d48232513d91c5d03d6769f9221db51f"
server_key_iv: "8ac5229775d687da450bbc2c64eb0027"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.042295 1692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.061312 1692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.063081 1692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.063190 1692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.063390 1692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:00.063691 1692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:00.063723 1692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.063745 1692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:00.063758 1692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.069350 1692 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34353
I20260619 10:24:00.069404 1823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:34353 every 8 connection(s)
I20260619 10:24:00.069708 1692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:24:00.075103 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1692
I20260619 10:24:00.075410 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:41813
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=35401
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:00.079001 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.070356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:54379 (local address 127.19.65.190:36015)
0619 10:24:00.070418 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.070420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.070651 (+ 231us) server_negotiation.cc:407] Connection header received
0619 10:24:00.071160 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.071161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.071176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.071226 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:00.072182 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.072657 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.073607 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.073810 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.073877 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.076123 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:00.076125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:00.076125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:00.076133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:00.077652 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.078057 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.078058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.078059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.078112 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.078348 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.078349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.078349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.078460 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:00.078526 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.078819 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.078905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:24:00.079159 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.070343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:54379)
0619 10:24:00.070530 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.070581 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.071054 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.071309 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.071313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.071865 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:00.072073 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.072078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.072744 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.072745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.073509 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.073511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.073894 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.073895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.074038 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.074477 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:00.074484 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:00.076029 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:00.077754 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:00.077761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:00.077763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:00.077973 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:00.078177 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:00.078178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:00.078178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:00.078281 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:00.078517 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:00.078523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:00.078651 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.078795 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.078940 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260619 10:24:00.079767 1824 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:24:00.079842 1824 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:00.080031 1824 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
I20260619 10:24:00.080415 1498 ts_manager.cc:194] Registered new tserver with Master: e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353)
I20260619 10:24:00.080969 1498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:54379
W20260619 10:24:00.155959 1827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.156137 1827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.156159 1827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.157223 1827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.157255 1827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.157280 1827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:24:00.158602 1827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=35401
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.158807 1827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.158970 1827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:24:00.164032 1827 server_base.cc:1034] running on GCE node
W20260619 10:24:00.164002 1833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.164016 1835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.164016 1832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.164366 1827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.164597 1827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.165753 1827 hybrid_clock.cc:648] HybridClock initialized: now 1781864640165743 us; error 29 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.168191 1827 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:24:00.168648 1827 webserver.cc:460] Webserver started at http://127.19.65.131:35401/ using document root <none> and password file <none>
I20260619 10:24:00.168812 1827 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.168859 1827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.170013 1827 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:00.170527 1842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.170693 1827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.170775 1827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "17150e9a2d19e338aedd38fac7d34241"
server_key_iv: "3e12830f749a0504e6e6308e36d6a1a8"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.171140 1827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.192677 1827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.194476 1827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.194583 1827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.194761 1827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:00.195070 1827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:00.195122 1827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.195156 1827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:00.195197 1827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.201088 1827 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41813
I20260619 10:24:00.201128 1955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:41813 every 8 connection(s)
I20260619 10:24:00.201491 1827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:00.210150 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.202143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:48053 (local address 127.19.65.190:36015)
0619 10:24:00.202202 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.202204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.202400 (+ 196us) server_negotiation.cc:407] Connection header received
0619 10:24:00.202897 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.202898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.202908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.202958 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:00.203918 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.204380 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.205285 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.205454 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.205499 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.207554 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:00.207555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:00.207555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:00.207561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:00.208991 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.209371 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.209372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.209372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.209424 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.209647 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.209648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.209648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.209743 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:00.209789 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.210020 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.210086 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:24:00.210433 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.202142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:48053)
0619 10:24:00.202288 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.202320 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.202807 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.203017 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.203020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.203632 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:00.203841 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.203846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.204452 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.204452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.205207 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.205209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.205517 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.205517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.205607 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.206015 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:00.206023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:00.207473 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:00.209067 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:00.209072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:00.209074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:00.209301 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:00.209499 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:00.209500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:00.209500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:00.209595 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:00.209802 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:00.209806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:00.209914 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.209998 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.210118 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260619 10:24:00.210574 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1827
I20260619 10:24:00.210964 1956 heartbeater.cc:347] Connected to a master server at 127.19.65.190:36015
I20260619 10:24:00.211063 1956 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:00.211244 1956 heartbeater.cc:511] Master 127.19.65.190:36015 requested a full tablet report, sending...
I20260619 10:24:00.211694 1498 ts_manager.cc:194] Registered new tserver with Master: 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
I20260619 10:24:00.212162 1498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:48053
I20260619 10:24:00.216935 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.210897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.19.65.190:36015)
0619 10:24:00.210942 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.210944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.211096 (+ 152us) server_negotiation.cc:407] Connection header received
0619 10:24:00.211177 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.211178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.211189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.211250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:00.211936 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.212526 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.213303 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.213475 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.213528 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.214575 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:00.214577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:00.214577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:00.214582 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:00.216065 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.216377 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.216378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.216378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.216424 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:00.216581 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:00.216581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:00.216581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:00.216664 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:00.216774 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.216823 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.216877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:24:00.217571 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:24:00.220391 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.217970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.19.65.190:36015)
0619 10:24:00.218030 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.218032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.218041 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:00.218111 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.218112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.218149 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.218226 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:24:00.218359 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.218817 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.219654 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.219855 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.219922 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.220086 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:24:00.220180 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:24:00.220266 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.220303 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.220320 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:24:00.220625 1498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:24:00.222231 1498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:24:00.224439 1620 tablet_service.cc:1467] Processing CreateTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 (DEFAULT_TABLE table=test-table [id=9bd9e2bfc71347f897bff5535727c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:00.224642 1620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08601af0cd7e4d5f907a70d1fbef2315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:00.226060 1578 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap starting.
I20260619 10:24:00.226377 1578 tablet_bootstrap.cc:654] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:00.226913 1578 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: No bootstrap required, opened a new log
I20260619 10:24:00.226965 1578 ts_tablet_manager.cc:1397] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 10:24:00.227080 1578 raft_consensus.cc:348] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.227160 1578 raft_consensus.cc:374] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:00.227209 1578 raft_consensus.cc:729] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877ae4a97e34df3b9df88be5c178d87, State: Initialized, Role: FOLLOWER
I20260619 10:24:00.227252 1578 consensus_queue.cc:260] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.227372 1578 ts_tablet_manager.cc:1428] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:00.228070 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.224213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.19.65.131:41813)
0619 10:24:00.224349 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.224353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.224380 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:24:00.224414 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.224416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.224430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.224505 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:00.224689 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.225828 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.226937 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.227745 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.227818 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.227877 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.227974 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.228013 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260619 10:24:00.228094 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.223991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:34353 (local address 127.0.0.1:49680)
0619 10:24:00.224089 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.224099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.224137 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.224500 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.224502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.224503 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:00.224541 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.224542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.225814 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.225815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.226928 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.226931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.227919 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.227920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.227978 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.227980 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.228007 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.228039 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 10:24:00.228089 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.224019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41813 (local address 127.0.0.1:32990)
0619 10:24:00.224113 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.224123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.224165 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.224579 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.224580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.224582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:00.224625 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.224627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.225944 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.225945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.226819 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.226822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.227832 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.227833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.227905 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.227909 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.227963 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.228009 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20260619 10:24:00.228338 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.224098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.19.65.130:34353)
0619 10:24:00.224309 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.224314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.224324 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:24:00.224360 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.224380 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.224395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.224503 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:00.224600 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.225698 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.227158 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.227825 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.228133 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.228199 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.228226 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.228268 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260619 10:24:00.229027 1758 tablet_service.cc:1467] Processing CreateTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 (DEFAULT_TABLE table=test-table [id=9bd9e2bfc71347f897bff5535727c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:00.229090 1890 tablet_service.cc:1467] Processing CreateTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 (DEFAULT_TABLE table=test-table [id=9bd9e2bfc71347f897bff5535727c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:00.229357 1890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08601af0cd7e4d5f907a70d1fbef2315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:00.229382 1758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08601af0cd7e4d5f907a70d1fbef2315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:00.232780 1965 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Bootstrap starting.
I20260619 10:24:00.233320 1965 tablet_bootstrap.cc:654] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:00.233815 1965 log.cc:824] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:00.234133 1966 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Bootstrap starting.
I20260619 10:24:00.234514 1965 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: No bootstrap required, opened a new log
I20260619 10:24:00.234567 1965 ts_tablet_manager.cc:1397] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:24:00.234679 1966 tablet_bootstrap.cc:654] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:00.235081 1966 log.cc:824] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:00.235850 1966 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: No bootstrap required, opened a new log
I20260619 10:24:00.235942 1966 ts_tablet_manager.cc:1397] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:24:00.236245 1965 raft_consensus.cc:348] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.236364 1965 raft_consensus.cc:374] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:00.236384 1965 raft_consensus.cc:729] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53f940be9494c068a6e8611aa0a7e25, State: Initialized, Role: FOLLOWER
I20260619 10:24:00.236505 1965 consensus_queue.cc:260] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.236784 1965 ts_tablet_manager.cc:1428] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:00.236802 1824 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
I20260619 10:24:00.237692 1966 raft_consensus.cc:348] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.237862 1966 raft_consensus.cc:374] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:00.237906 1966 raft_consensus.cc:729] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fafdc5a9514421b3af4c5b391d0ca9, State: Initialized, Role: FOLLOWER
I20260619 10:24:00.238030 1966 consensus_queue.cc:260] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.238286 1966 ts_tablet_manager.cc:1428] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:24:00.238320 1956 heartbeater.cc:503] Master 127.19.65.190:36015 was elected leader, sending a full tablet report...
W20260619 10:24:00.320179 1825 tablet.cc:2369] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:24:00.420477 1969 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:00.420595 1969 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.420955 1969 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813), 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
W20260619 10:24:00.421382 1687 tablet.cc:2369] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:24:00.423902 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.421297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:56703 (local address 127.19.65.129:40905)
0619 10:24:00.421355 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.421357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.421478 (+ 121us) server_negotiation.cc:407] Connection header received
0619 10:24:00.421550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.421551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.421561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.421609 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:00.421777 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.422265 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423174 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.423553 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423611 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.423639 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.423778 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.423822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:24:00.423929 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.421224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:49959 (local address 127.19.65.131:41813)
0619 10:24:00.421305 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.421308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.421317 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:00.421345 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.421346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.421354 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.421410 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:00.421617 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.422143 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423093 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.423565 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423650 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.423658 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.423830 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.423867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 10:24:00.423920 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.421246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.19.65.130:56703)
0619 10:24:00.421424 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.421434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.421485 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.421637 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.421638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.421640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:00.421709 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.421711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.422343 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.422344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.423094 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.423095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.423633 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.423708 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.423711 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.423753 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.423782 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
I20260619 10:24:00.423923 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.421105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41813 (local address 127.19.65.130:49959)
0619 10:24:00.421159 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.421172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.421215 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.421440 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:00.421442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:00.421444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:00.421515 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.421517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.422231 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.422232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.422999 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:00.423001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.423668 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.423669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:00.423734 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.423751 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 10:24:00.423809 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:24:00.423845 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:24:00.424279 1640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08601af0cd7e4d5f907a70d1fbef2315" candidate_uuid: "e53f940be9494c068a6e8611aa0a7e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877ae4a97e34df3b9df88be5c178d87" is_pre_election: true
I20260619 10:24:00.424458 1640 raft_consensus.cc:2454] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e53f940be9494c068a6e8611aa0a7e25 in term 0.
I20260619 10:24:00.424551 1910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08601af0cd7e4d5f907a70d1fbef2315" candidate_uuid: "e53f940be9494c068a6e8611aa0a7e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" is_pre_election: true
I20260619 10:24:00.424701 1910 raft_consensus.cc:2454] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e53f940be9494c068a6e8611aa0a7e25 in term 0.
I20260619 10:24:00.424696 1713 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87, e53f940be9494c068a6e8611aa0a7e25; no voters:
I20260619 10:24:00.424803 1969 raft_consensus.cc:2781] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:24:00.424862 1969 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:24:00.424903 1969 raft_consensus.cc:3037] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:00.425585 1969 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.425707 1969 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [CANDIDATE]: Term 1 election: Requested vote from peers 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813), 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905)
I20260619 10:24:00.425875 1640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08601af0cd7e4d5f907a70d1fbef2315" candidate_uuid: "e53f940be9494c068a6e8611aa0a7e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877ae4a97e34df3b9df88be5c178d87"
I20260619 10:24:00.425894 1910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08601af0cd7e4d5f907a70d1fbef2315" candidate_uuid: "e53f940be9494c068a6e8611aa0a7e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
I20260619 10:24:00.425958 1910 raft_consensus.cc:3037] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:00.425958 1640 raft_consensus.cc:3037] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:00.426891 1910 raft_consensus.cc:2454] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53f940be9494c068a6e8611aa0a7e25 in term 1.
I20260619 10:24:00.427055 1713 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25; no voters:
I20260619 10:24:00.427140 1640 raft_consensus.cc:2454] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53f940be9494c068a6e8611aa0a7e25 in term 1.
I20260619 10:24:00.427157 1969 raft_consensus.cc:2781] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:24:00.427279 1969 raft_consensus.cc:686] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 1 LEADER]: Becoming Leader. State: Replica: e53f940be9494c068a6e8611aa0a7e25, State: Running, Role: LEADER
I20260619 10:24:00.427385 1969 consensus_queue.cc:237] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.428121 1498 catalog_manager.cc:5495] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 reported cstate change: term changed from 0 to 1, leader changed from <none> to e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "e53f940be9494c068a6e8611aa0a7e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } health_report { overall_health: UNKNOWN } } }
W20260619 10:24:00.451992 1957 tablet.cc:2369] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:24:00.452515 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.450133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.19.65.130:34353)
0619 10:24:00.450193 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:00.450195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:00.450204 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:00.450262 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:00.450263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:00.450286 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:00.450355 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:24:00.450494 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.450952 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.451723 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:00.451938 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:00.452009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:00.452156 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:24:00.452261 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:24:00.452339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:00.452392 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:24:00.452411 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:24:00.454228 1910 raft_consensus.cc:1261] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Refusing update from remote peer e53f940be9494c068a6e8611aa0a7e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:00.454553 1640 raft_consensus.cc:1261] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Refusing update from remote peer e53f940be9494c068a6e8611aa0a7e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:00.454555 1972 consensus_queue.cc:1035] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:00.454839 1972 consensus_queue.cc:1035] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:00.456758 1975 mvcc.cc:205] Tried to move back new op lower bound from 7298517567297941504 to 7298517567191113728. Current Snapshot: MvccSnapshot[applied={T|T < 7298517567297941504}]
I20260619 10:24:00.457176 1978 mvcc.cc:205] Tried to move back new op lower bound from 7298517567297941504 to 7298517567191113728. Current Snapshot: MvccSnapshot[applied={T|T < 7298517567297941504}]
I20260619 10:24:00.459090 1497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:00.459189 1497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:00.459889 1497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Sending DeleteTablet for 3 replicas of tablet 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:00.460156 1758 tablet_service.cc:1514] Processing DeleteTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:49680
I20260619 10:24:00.460250 1890 tablet_service.cc:1514] Processing DeleteTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:32990
I20260619 10:24:00.460279 1620 tablet_service.cc:1514] Processing DeleteTablet for tablet 08601af0cd7e4d5f907a70d1fbef2315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:00 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:50338
I20260619 10:24:00.460454 1986 tablet_replica.cc:330] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: stopping tablet replica
I20260619 10:24:00.460464 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1536
I20260619 10:24:00.461403 1985 tablet_replica.cc:330] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: stopping tablet replica
I20260619 10:24:00.461510 1985 raft_consensus.cc:2229] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:24:00.461632 1985 raft_consensus.cc:2258] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:00.462386 1985 ts_tablet_manager.cc:1905] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:00.464457 1985 ts_tablet_manager.cc:1918] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:00.464531 1985 log.cc:1196] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:00.464807 1985 ts_tablet_manager.cc:1939] T 08601af0cd7e4d5f907a70d1fbef2315 P e53f940be9494c068a6e8611aa0a7e25: Deleting consensus metadata
I20260619 10:24:00.465184 1482 catalog_manager.cc:4841] TS e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): tablet 08601af0cd7e4d5f907a70d1fbef2315 (table test-table [id=9bd9e2bfc71347f897bff5535727c382]) successfully deleted
W20260619 10:24:00.466948 1484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40905 (error 108)
I20260619 10:24:00.467273 1987 tablet_replica.cc:330] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: stopping tablet replica
I20260619 10:24:00.467434 1987 raft_consensus.cc:2229] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:24:00.467550 1987 raft_consensus.cc:2258] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:00.467765 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1692
I20260619 10:24:00.467983 1987 ts_tablet_manager.cc:1905] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:00.469406 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.469242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40905 (local address 127.0.0.1:50344)
0619 10:24:00.469305 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.469345 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 10:24:00.469655 1987 ts_tablet_manager.cc:1918] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:00.469726 1987 log.cc:1196] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:00.469547 1484 catalog_manager.cc:4584] TS 7877ae4a97e34df3b9df88be5c178d87 (127.19.65.129:40905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08601af0cd7e4d5f907a70d1fbef2315: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40905: connect: Connection refused (error 111)
I20260619 10:24:00.469990 1987 ts_tablet_manager.cc:1939] T 08601af0cd7e4d5f907a70d1fbef2315 P 20fafdc5a9514421b3af4c5b391d0ca9: Deleting consensus metadata
I20260619 10:24:00.470366 1484 catalog_manager.cc:4841] TS 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): tablet 08601af0cd7e4d5f907a70d1fbef2315 (table test-table [id=9bd9e2bfc71347f897bff5535727c382]) successfully deleted
I20260619 10:24:00.473119 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1827
I20260619 10:24:00.477885 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 1465
I20260619 10:24:00.483043 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--webserver_interface=127.19.65.190
--webserver_port=37999
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:00.562253 1989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.562451 1989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.562474 1989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.563577 1989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:24:00.563612 1989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.563625 1989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.563637 1989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:24:00.563689 1989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:24:00.565104 1989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.565341 1989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.565564 1989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:00.570327 1994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.570266 1997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.570367 1989 server_base.cc:1034] running on GCE node
W20260619 10:24:00.570266 1995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.570797 1989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.571053 1989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.572187 1989 hybrid_clock.cc:648] HybridClock initialized: now 1781864640572178 us; error 28 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.574404 1989 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:24:00.574781 1989 webserver.cc:460] Webserver started at http://127.19.65.190:37999/ using document root <none> and password file <none>
I20260619 10:24:00.574990 1989 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.575052 1989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.576318 1989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:00.576962 2004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.577122 1989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:00.577211 1989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "ae1856a171da99a3f758a0fac8820635"
server_key_iv: "a552ed660e4e7dd6be6407f988048403"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.577543 1989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.589346 1989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.591073 1989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.591216 1989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.595012 1989 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36015
I20260619 10:24:00.595060 2056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36015 every 8 connection(s)
I20260619 10:24:00.595476 1989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:24:00.596937 2057 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:24:00.597189 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 1989
I20260619 10:24:00.597468 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:40905
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=42945
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:00.597855 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap starting.
I20260619 10:24:00.606227 2057 log.cc:824] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:00.608309 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:00.608585 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e: Bootstrap complete.
I20260619 10:24:00.610651 2057 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:24:00.610886 2057 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Initialized, Role: FOLLOWER
I20260619 10:24:00.611061 2057 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:24:00.611116 2057 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:24:00.611167 2057 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:24:00.611212 2057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 2 FOLLOWER]: Advancing to term 3
I20260619 10:24:00.612159 2057 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:24:00.612335 2057 leader_election.cc:302] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45bdd17bea14c8ba8edc23aa88f2d5e; no voters:
I20260619 10:24:00.612591 2057 leader_election.cc:288] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 10:24:00.612627 2061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 3 FOLLOWER]: Leader election won for term 3
I20260619 10:24:00.612764 2061 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [term 3 LEADER]: Becoming Leader. State: Replica: b45bdd17bea14c8ba8edc23aa88f2d5e, State: Running, Role: LEADER
I20260619 10:24:00.612895 2057 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:24:00.612864 2061 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } }
I20260619 10:24:00.613205 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45bdd17bea14c8ba8edc23aa88f2d5e. Latest consensus state: current_term: 3 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:24:00.613220 2062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45bdd17bea14c8ba8edc23aa88f2d5e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36015 } } }
I20260619 10:24:00.613371 2062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:00.613736 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45bdd17bea14c8ba8edc23aa88f2d5e [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:00.613845 2069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:24:00.614583 2069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:24:00.614679 2069 catalog_manager.cc:1259] Loaded cluster ID: 803a7867884a4c5283b95b167b83a921
I20260619 10:24:00.614711 2069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 10:24:00.615025 2069 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18092ef9ad google::LogMessage::Fail() at ??:0
@ 0x7f18092f288a google::LogMessage::SendToLog() at ??:0
@ 0x7f18092ef4de google::LogMessage::Flush() at ??:0
@ 0x7f18092f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18112100f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f18112198b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f180b1bd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f180b1b2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f180c0166db start_thread at ??:0
@ 0x7f1807ded61f clone at ??:0
W20260619 10:24:00.689705 2059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.689915 2059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.689951 2059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.691068 2059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.691128 2059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.691171 2059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:24:00.692716 2059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:40905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=42945
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.692966 2059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.693176 2059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:00.698007 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.698029 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.698182 2059 server_base.cc:1034] running on GCE node
W20260619 10:24:00.698138 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.698423 2059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.698671 2059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.699810 2059 hybrid_clock.cc:648] HybridClock initialized: now 1781864640699799 us; error 25 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.702219 2059 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:24:00.702669 2059 webserver.cc:460] Webserver started at http://127.19.65.129:42945/ using document root <none> and password file <none>
I20260619 10:24:00.702879 2059 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.702946 2059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.704329 2059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:00.705063 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.705245 2059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:00.705348 2059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "7877ae4a97e34df3b9df88be5c178d87"
format_stamp: "Formatted at 2026-06-19 10:23:58 on dist-test-slave-xd2q"
server_key: "2247090af2c6683886d612e3d2027e01"
server_key_iv: "fbcd7e3c736ea1d91246ee65ba91dfdf"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.705670 2059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.722523 2059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.724746 2059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.724889 2059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.725109 2059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:00.725529 2110 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:24:00.727408 2059 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:24:00.727465 2059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:24:00.727507 2059 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:24:00.728103 2059 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:24:00.728155 2059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:24:00.728217 2110 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap starting.
I20260619 10:24:00.735258 2059 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40905
I20260619 10:24:00.735621 2059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:24:00.738930 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40905 every 8 connection(s)
I20260619 10:24:00.739125 2110 log.cc:824] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:00.741762 2110 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:00.742074 2110 tablet_bootstrap.cc:492] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Bootstrap complete.
I20260619 10:24:00.742627 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2059
I20260619 10:24:00.742854 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:34353
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=38443
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:00.743359 2110 ts_tablet_manager.cc:1397] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260619 10:24:00.744699 2110 raft_consensus.cc:348] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.745036 2110 raft_consensus.cc:729] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877ae4a97e34df3b9df88be5c178d87, State: Initialized, Role: FOLLOWER
I20260619 10:24:00.745180 2110 consensus_queue.cc:260] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:00.745434 2110 ts_tablet_manager.cc:1428] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260619 10:24:00.800132 2220 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:24:00.736261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:60871)
0619 10:24:00.736418 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.736467 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:00.736952 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:00.799950 (+ 62998us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
W20260619 10:24:00.800401 2218 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:36015 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:36015: Client connection negotiation failed: client connection to 127.19.65.190:36015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:24:00.800742 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.800631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:48947)
0619 10:24:00.800678 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.800713 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:24:00.801105 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.801030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:57601)
0619 10:24:00.801070 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.801091 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:24:00.801213 2218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:24:00.823602 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.823779 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.823802 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.824973 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.825011 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.825060 2222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:24:00.826493 2222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:34353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=38443
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.826725 2222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.826936 2222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:00.831683 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.831715 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.831890 2222 server_base.cc:1034] running on GCE node
W20260619 10:24:00.832024 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.832254 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.832562 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.833739 2222 hybrid_clock.cc:648] HybridClock initialized: now 1781864640833713 us; error 43 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.836961 2222 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:24:00.837417 2222 webserver.cc:460] Webserver started at http://127.19.65.130:38443/ using document root <none> and password file <none>
I20260619 10:24:00.837679 2222 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.837740 2222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.839262 2222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:24:00.839833 2238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.839998 2222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.840099 2222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "e53f940be9494c068a6e8611aa0a7e25"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "d48232513d91c5d03d6769f9221db51f"
server_key_iv: "8ac5229775d687da450bbc2c64eb0027"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.840415 2222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.856755 2222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.859045 2222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.859191 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.859447 2222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:00.859750 2222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:00.859802 2222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.859843 2222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:00.859872 2222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.865661 2222 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34353
I20260619 10:24:00.865708 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:34353 every 8 connection(s)
I20260619 10:24:00.866019 2222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:24:00.866966 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.866614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:55375)
0619 10:24:00.866756 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.866828 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
W20260619 10:24:00.867348 2352 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:36015 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:36015: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
I20260619 10:24:00.867586 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.867522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:48267)
0619 10:24:00.867548 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.867571 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260619 10:24:00.867846 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.867753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:42213)
0619 10:24:00.867796 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.867823 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:24:00.867965 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2222
W20260619 10:24:00.868000 2352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:24:00.868220 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:41813
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:36015
--webserver_port=35401
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:00.949101 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:00.949288 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:00.949317 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:00.950485 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:00.950524 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:00.950572 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:24:00.952031 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=35401
--tserver_master_addrs=127.19.65.190:36015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:00.952239 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:00.952417 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:00.957156 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.957156 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:00.957204 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:00.957320 2355 server_base.cc:1034] running on GCE node
I20260619 10:24:00.957527 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:00.957759 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:00.958921 2355 hybrid_clock.cc:648] HybridClock initialized: now 1781864640958907 us; error 24 us; skew 500 ppm
Jun 19 10:24:00 dist-test-slave-xd2q krb5kdc[921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:00.961267 2355 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:24:00.961702 2355 webserver.cc:460] Webserver started at http://127.19.65.131:35401/ using document root <none> and password file <none>
I20260619 10:24:00.961910 2355 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:00.961972 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:00.963253 2355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:24:00.963852 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.964033 2355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.964124 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "20fafdc5a9514421b3af4c5b391d0ca9"
format_stamp: "Formatted at 2026-06-19 10:23:59 on dist-test-slave-xd2q"
server_key: "17150e9a2d19e338aedd38fac7d34241"
server_key_iv: "3e12830f749a0504e6e6308e36d6a1a8"
server_key_version: "encryptionkey@0"
I20260619 10:24:00.964459 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:00.977614 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:00.979240 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:00.979423 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:00.979621 2355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:00.979897 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:00.979946 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.979985 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:00.980010 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:00.985637 2355 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41813
I20260619 10:24:00.985680 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:41813 every 8 connection(s)
I20260619 10:24:00.985997 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:24:00.986994 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.986624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:58251)
0619 10:24:00.986796 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.986867 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
W20260619 10:24:00.987411 2484 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:36015 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:36015: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
I20260619 10:24:00.987677 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.987597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:53743)
0619 10:24:00.987637 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.987661 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:24:00.987910 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:00.987865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:38845)
0619 10:24:00.987883 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:00.987899 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260619 10:24:00.988039 2484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:24:00.992688 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2355
I20260619 10:24:01.801740 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:01.801530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:58411)
0619 10:24:01.801666 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:01.801695 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260619 10:24:01.868438 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:01.868251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:54013)
0619 10:24:01.868354 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:01.868391 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260619 10:24:01.988476 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:01.988303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:60283)
0619 10:24:01.988402 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:01.988431 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
I20260619 10:24:02.172619 2491 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:02.172734 2491 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:02.173035 2491 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
Jun 19 10:24:02 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.130@KRBTEST.COM
Jun 19 10:24:02 dist-test-slave-xd2q krb5kdc[921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864640, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.131@KRBTEST.COM
I20260619 10:24:02.182919 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.173185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41813 (local address 127.19.65.129:49245)
0619 10:24:02.173333 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:02.173345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:02.173400 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:02.174087 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:02.174088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:02.174407 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:02.174583 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:02.174589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.175728 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.175729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:02.176461 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:02.176462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.177027 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.177028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:02.177134 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:02.177681 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:02.177688 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:02.179594 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:02.181376 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:02.181377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:02.181377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:02.181693 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:02.182001 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:02.182002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:02.182002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:02.182077 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:02.182523 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:02.182523 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:02.182581 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 10:24:02.182659 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:24:02.182817 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":7,"thread_start_us":48,"threads_started":1}
I20260619 10:24:02.183028 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.173308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:49245 (local address 127.19.65.131:41813)
0619 10:24:02.173465 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:02.173474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:02.173505 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:24:02.173924 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:02.173926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:02.173970 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:02.174106 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:02.174722 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.175630 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.176559 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.176944 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.177056 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:02.179690 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:02.179693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:02.179693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:02.179704 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:02.181293 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:02.181790 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:02.181793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:02.181793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:02.181934 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:02.182148 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:02.182149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:02.182150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:02.182446 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:02.182527 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:02.182787 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 10:24:02.182917 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260619 10:24:02.183084 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.173230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:54059 (local address 127.19.65.130:34353)
0619 10:24:02.173346 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:02.173357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:02.173394 (+ 37us) server_negotiation.cc:407] Connection header received
0619 10:24:02.173822 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:02.173824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:02.173862 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:02.174026 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:02.174719 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.175619 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.176561 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.176953 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.177073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:02.179280 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:02.179284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:02.179285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:02.179337 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:02.181183 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:02.181646 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:02.181649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:02.181650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:02.181793 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:02.182071 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:02.182072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:02.182072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:02.182374 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:02.182437 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:02.182834 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 10:24:02.182957 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260619 10:24:02.183025 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.173130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:34353 (local address 127.19.65.129:54059)
0619 10:24:02.173180 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:02.173190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:02.173244 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:02.174061 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:02.174063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:02.174405 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:02.174583 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:02.174589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.175712 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.175713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:02.176460 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:02.176463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:02.177044 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:02.177045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:02.177136 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:02.177740 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:02.177745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:02.179165 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:02.181262 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:02.181268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:02.181270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:02.181564 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:02.181877 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:02.181877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:02.181878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:02.182006 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:02.182439 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:02.182445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:02.182581 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:24:02.182755 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 10:24:02.182896 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":77}
W20260619 10:24:02.184315 2104 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:02.184401 2106 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:02.184494 2106 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87; no voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25
I20260619 10:24:02.184592 2491 raft_consensus.cc:2726] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:24:02.802448 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.802219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:35285)
0619 10:24:02.802369 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:02.802402 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260619 10:24:02.869014 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.868847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:44093)
0619 10:24:02.868949 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:02.868980 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260619 10:24:02.989075 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:02.988881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:59235)
0619 10:24:02.989002 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:02.989036 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260619 10:24:03.803046 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:03.802852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:41447)
0619 10:24:03.802971 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:03.803002 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260619 10:24:03.869616 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:03.869421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:44987)
0619 10:24:03.869547 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:03.869580 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260619 10:24:03.917881 2500 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:03.917968 2500 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:03.918180 2500 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
W20260619 10:24:03.918664 2104 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:03.918851 2106 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:03.918921 2106 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87; no voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25
I20260619 10:24:03.919014 2500 raft_consensus.cc:2726] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:24:03.989629 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:03.989459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:42751)
0619 10:24:03.989564 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:03.989596 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260619 10:24:04.803687 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:04.803490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:37823)
0619 10:24:04.803601 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:04.803642 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260619 10:24:04.870244 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:04.870031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:44291)
0619 10:24:04.870144 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:04.870178 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260619 10:24:04.990197 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:04.990013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:47493)
0619 10:24:04.990130 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:04.990163 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260619 10:24:05.804324 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:05.804123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:52183)
0619 10:24:05.804236 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:05.804281 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260619 10:24:05.870786 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:05.870624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:46459)
0619 10:24:05.870722 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:05.870753 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
I20260619 10:24:05.990844 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:05.990636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:58107)
0619 10:24:05.990775 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:05.990809 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260619 10:24:06.475873 2508 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:06.475981 2508 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:06.476186 2508 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
W20260619 10:24:06.476542 2104 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:06.476616 2106 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:06.476718 2106 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87; no voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25
I20260619 10:24:06.476800 2508 raft_consensus.cc:2726] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:24:06.805075 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:06.804812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:54767)
0619 10:24:06.804927 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:06.805009 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260619 10:24:06.871438 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:06.871180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:60665)
0619 10:24:06.871366 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:06.871401 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":52,"threads_started":1}
I20260619 10:24:06.991542 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:06.991376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:42353)
0619 10:24:06.991477 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:06.991509 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260619 10:24:07.805727 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:07.805482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:43369)
0619 10:24:07.805629 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:07.805683 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260619 10:24:07.872032 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:07.871844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:58759)
0619 10:24:07.871964 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:07.871997 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260619 10:24:07.992126 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:07.991934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:38201)
0619 10:24:07.992054 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:07.992089 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260619 10:24:08.806463 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:08.806229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:41175)
0619 10:24:08.806379 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:08.806415 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260619 10:24:08.872529 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:08.872367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:47619)
0619 10:24:08.872466 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:08.872495 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260619 10:24:08.992803 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:08.992590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:52075)
0619 10:24:08.992733 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:08.992768 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260619 10:24:09.422960 2518 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:09.423059 2518 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:09.423278 2518 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
W20260619 10:24:09.423627 2106 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:09.423727 2104 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:09.423804 2104 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87; no voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25
I20260619 10:24:09.423898 2518 raft_consensus.cc:2726] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:24:09.807159 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:09.806920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:57833)
0619 10:24:09.807078 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:09.807112 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260619 10:24:09.873050 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:09.872879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:52013)
0619 10:24:09.872987 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:09.873018 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260619 10:24:09.993429 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:09.993232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:38777)
0619 10:24:09.993353 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:09.993392 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260619 10:24:10.807906 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:10.807647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:50043)
0619 10:24:10.807799 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:10.807849 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260619 10:24:10.873628 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:10.873445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:58867)
0619 10:24:10.873563 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:10.873594 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260619 10:24:10.994117 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:10.993914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:51215)
0619 10:24:10.994049 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:10.994082 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260619 10:24:11.808511 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:11.808304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:60931)
0619 10:24:11.808432 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:11.808465 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260619 10:24:11.874276 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:11.874071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:44871)
0619 10:24:11.874202 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:11.874235 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":98,"threads_started":1}
I20260619 10:24:11.994751 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:11.994553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:56515)
0619 10:24:11.994684 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:11.994717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260619 10:24:12.809100 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:12.808906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:36409)
0619 10:24:12.809022 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:12.809056 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260619 10:24:12.874886 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:12.874700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:57503)
0619 10:24:12.874819 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:12.874850 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260619 10:24:12.995558 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:12.995226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:52175)
0619 10:24:12.995493 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:12.995523 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":214,"threads_started":1}
I20260619 10:24:13.809859 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:13.809623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:42923)
0619 10:24:13.809764 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:13.809814 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260619 10:24:13.875528 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:13.875326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:47991)
0619 10:24:13.875462 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:13.875493 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260619 10:24:13.996140 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:13.995952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:38919)
0619 10:24:13.996074 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:13.996106 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260619 10:24:14.810393 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:14.810197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:49417)
0619 10:24:14.810322 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:14.810351 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260619 10:24:14.876075 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:14.875895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:49657)
0619 10:24:14.876004 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:14.876037 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260619 10:24:14.996721 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:14.996548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:38949)
0619 10:24:14.996654 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:14.996685 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260619 10:24:15.022887 2537 raft_consensus.cc:482] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:15.022972 2537 raft_consensus.cc:504] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53f940be9494c068a6e8611aa0a7e25" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34353 } } peers { permanent_uuid: "20fafdc5a9514421b3af4c5b391d0ca9" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41813 } } peers { permanent_uuid: "7877ae4a97e34df3b9df88be5c178d87" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40905 } }
I20260619 10:24:15.023170 2537 leader_election.cc:288] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353), 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813)
W20260619 10:24:15.023574 2104 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e53f940be9494c068a6e8611aa0a7e25 (127.19.65.130:34353): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
W20260619 10:24:15.023718 2106 leader_election.cc:341] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20fafdc5a9514421b3af4c5b391d0ca9 (127.19.65.131:41813): Not found: Tablet not found: 08601af0cd7e4d5f907a70d1fbef2315
I20260619 10:24:15.023802 2106 leader_election.cc:302] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877ae4a97e34df3b9df88be5c178d87; no voters: 20fafdc5a9514421b3af4c5b391d0ca9, e53f940be9494c068a6e8611aa0a7e25
I20260619 10:24:15.023895 2537 raft_consensus.cc:2726] T 08601af0cd7e4d5f907a70d1fbef2315 P 7877ae4a97e34df3b9df88be5c178d87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:24:15.811108 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:15.810903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.129:49555)
0619 10:24:15.811030 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:15.811064 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260619 10:24:15.876675 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:15.876495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.130:44053)
0619 10:24:15.876590 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:15.876618 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260619 10:24:15.995788 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2059
I20260619 10:24:15.997363 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:15.997195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:36015 (local address 127.19.65.131:34103)
0619 10:24:15.997298 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:15.997330 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260619 10:24:16.000384 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2222
I20260619 10:24:16.004268 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2355
2026-06-19T10:24:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18526 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:24:16 dist-test-slave-xd2q krb5kdc[2543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:24:16 dist-test-slave-xd2q krb5kdc[2543](info): set up 2 sockets
Jun 19 10:24:16 dist-test-slave-xd2q krb5kdc[2543](info): commencing operation
krb5kdc: starting...
W20260619 10:24:17.023116 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:24:17.130481 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:17.207227 2556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:17.207475 2556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:17.207512 2556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:17.208648 2556 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:24:17.208712 2556 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:24:17.208734 2556 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:24:17.208755 2556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:24:17.208786 2556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:17.208810 2556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:17.208838 2556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:24:17.208868 2556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:24:17.210266 2556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:17.210561 2556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:17.210759 2556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:17.215479 2561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.215493 2564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.215497 2562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:17.215706 2556 server_base.cc:1034] running on GCE node
I20260619 10:24:17.215920 2556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:17.216128 2556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:17.217262 2556 hybrid_clock.cc:648] HybridClock initialized: now 1781864657217246 us; error 28 us; skew 500 ppm
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:17.219558 2556 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:24:17.219964 2556 webserver.cc:460] Webserver started at http://127.19.65.190:43409/ using document root <none> and password file <none>
I20260619 10:24:17.220170 2556 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:17.220213 2556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:17.220350 2556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:24:17.221318 2556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "230de586ac9b423c9834e331c1f1dfc8"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "607bd41616e3fe7bbb5e537b12cabd18"
server_key_iv: "e3fa24c749cec89ac193d5ceaf459226"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.221650 2556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "230de586ac9b423c9834e331c1f1dfc8"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "607bd41616e3fe7bbb5e537b12cabd18"
server_key_iv: "e3fa24c749cec89ac193d5ceaf459226"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.222877 2556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:24:17.223596 2571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.223766 2556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:17.223870 2556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "230de586ac9b423c9834e331c1f1dfc8"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "607bd41616e3fe7bbb5e537b12cabd18"
server_key_iv: "e3fa24c749cec89ac193d5ceaf459226"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.223932 2556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:17.231657 2556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:17.233217 2556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:17.233314 2556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:17.236867 2556 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43483
I20260619 10:24:17.236896 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:43483 every 8 connection(s)
I20260619 10:24:17.237190 2556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:24:17.238027 2624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:17.240182 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap starting.
I20260619 10:24:17.240756 2624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:17.241046 2624 log.cc:824] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:17.241672 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: No bootstrap required, opened a new log
I20260619 10:24:17.243180 2624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:17.243428 2624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:17.243489 2624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Initialized, Role: FOLLOWER
I20260619 10:24:17.243645 2624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:17.243705 2624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:24:17.243723 2624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:24:17.243788 2624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:17.244223 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2556
I20260619 10:24:17.244336 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:24:17.244331 2624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:17.244455 2624 leader_election.cc:302] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230de586ac9b423c9834e331c1f1dfc8; no voters:
I20260619 10:24:17.244570 19718 external_mini_cluster.cc:1428] Setting key 4a51fe3c3cc9d4519174795138e09732
I20260619 10:24:17.244632 2624 leader_election.cc:288] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:24:17.244661 2627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:24:17.244777 2627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 LEADER]: Becoming Leader. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Running, Role: LEADER
I20260619 10:24:17.244879 2624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:24:17.244947 2627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:17.245584 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:17.245767 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:17.245595 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230de586ac9b423c9834e331c1f1dfc8. Latest consensus state: current_term: 1 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:17.246351 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
W20260619 10:24:17.246853 2644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:24:17.246908 2644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:24:17.246999 2638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:24:17.247179 2638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:24:17.248682 2638 catalog_manager.cc:1347] Generated new cluster ID: 334fa5febb664ae4b251fc2fbdd85df9
I20260619 10:24:17.248734 2638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:17.256893 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.245192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.19.65.190:43483)
0619 10:24:17.245370 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.245379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.245409 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:24:17.245530 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.245537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.245593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.245769 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:17.246558 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.247853 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.249013 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.249589 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.249683 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.252626 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:17.252631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:17.252631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:17.252643 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:17.254940 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.255372 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.255375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.255376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.255507 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.255796 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.255797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.255798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.256121 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:17.256307 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.256419 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.256682 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260619 10:24:17.264650 2638 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:24:17.265349 2638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:24:17.274866 2638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Generated new TSK 0
I20260619 10:24:17.275043 2638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:24:17.302800 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:17.380899 2650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:17.381107 2650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:17.381141 2650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:17.382234 2650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:17.382318 2650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:17.382364 2650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:24:17.383816 2650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:17.384057 2650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:17.384270 2650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:17.389122 2658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.389132 2656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:17.389389 2650 server_base.cc:1034] running on GCE node
W20260619 10:24:17.389148 2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:17.389633 2650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:17.389871 2650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:17.391031 2650 hybrid_clock.cc:648] HybridClock initialized: now 1781864657391026 us; error 35 us; skew 500 ppm
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:17.393362 2650 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:24:17.393747 2650 webserver.cc:460] Webserver started at http://127.19.65.129:38637/ using document root <none> and password file <none>
I20260619 10:24:17.393999 2650 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:17.394066 2650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:17.394212 2650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:24:17.395192 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "731285bf624a4544bd96b020a65449f7"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "ade113217354518cbecdb5b1a69bf5f7"
server_key_iv: "9d9e68d8504955d78678d6aaae55fbf3"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.395607 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "731285bf624a4544bd96b020a65449f7"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "ade113217354518cbecdb5b1a69bf5f7"
server_key_iv: "9d9e68d8504955d78678d6aaae55fbf3"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.396823 2650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:17.397538 2665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.397711 2650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:17.397805 2650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "731285bf624a4544bd96b020a65449f7"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "ade113217354518cbecdb5b1a69bf5f7"
server_key_iv: "9d9e68d8504955d78678d6aaae55fbf3"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.397888 2650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:17.431337 2650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:17.433025 2650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:17.433171 2650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:17.433403 2650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:17.433681 2650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:17.433738 2650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.433779 2650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:17.433806 2650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.439432 2650 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40083
I20260619 10:24:17.439519 2778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40083 every 8 connection(s)
I20260619 10:24:17.439814 2650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:17.447680 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2650
I20260619 10:24:17.447798 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:24:17.447999 19718 external_mini_cluster.cc:1428] Setting key 87cb390b597e7ba694e79f9b8cb1dfdd
I20260619 10:24:17.449146 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.440479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36763 (local address 127.19.65.190:43483)
0619 10:24:17.440519 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.440520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.440681 (+ 161us) server_negotiation.cc:407] Connection header received
0619 10:24:17.441124 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.441125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.441137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.441183 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:17.442001 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.442488 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.443476 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.443676 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.443726 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.445862 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:17.445864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:17.445864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:17.445874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:17.447500 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.447979 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.447980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.447981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.448040 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.448491 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.448492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.448492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.448639 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:17.448699 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.448979 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.449063 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:24:17.449361 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.440445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:36763)
0619 10:24:17.440586 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.440616 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.441041 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.441231 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.441234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.441758 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:17.441925 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.441930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.442562 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.442563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.443391 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.443393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.443752 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.443753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.443844 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.444262 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:17.444269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:17.445767 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:17.447591 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.447598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.447600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.447884 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.448125 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.448126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.448126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.448259 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.448703 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:17.448710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:17.448841 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.448944 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.449087 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260619 10:24:17.449760 2779 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:17.449852 2779 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:17.450016 2779 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:17.450485 2588 ts_manager.cc:194] Registered new tserver with Master: 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:17.451053 2588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:36763
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:24:17.471514 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:17.548740 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:17.548908 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:17.548931 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:17.550035 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:17.550074 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:17.550099 2784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:24:17.551553 2784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:17.551779 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:17.551988 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:17.556638 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.556656 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.556651 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:17.556685 2784 server_base.cc:1034] running on GCE node
I20260619 10:24:17.557035 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:17.557298 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:17.558470 2784 hybrid_clock.cc:648] HybridClock initialized: now 1781864657558461 us; error 39 us; skew 500 ppm
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:17.561422 2784 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:24:17.561865 2784 webserver.cc:460] Webserver started at http://127.19.65.130:37335/ using document root <none> and password file <none>
I20260619 10:24:17.562100 2784 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:17.562165 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:17.562325 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:24:17.563283 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "a8fc42d58989417b9fcc29e868c8a2cc"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "554d7fbbc15413db786ba6d7a92aef57"
server_key_iv: "0a93819b94248effcd19f256b037fe3e"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.563690 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "a8fc42d58989417b9fcc29e868c8a2cc"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "554d7fbbc15413db786ba6d7a92aef57"
server_key_iv: "0a93819b94248effcd19f256b037fe3e"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.564862 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:24:17.565518 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.565688 2784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:24:17.565778 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "a8fc42d58989417b9fcc29e868c8a2cc"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "554d7fbbc15413db786ba6d7a92aef57"
server_key_iv: "0a93819b94248effcd19f256b037fe3e"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.565860 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:17.586920 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:17.589171 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:17.589325 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:17.589547 2784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:17.589854 2784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:17.589903 2784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.589943 2784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:17.589972 2784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.595717 2784 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:39737
I20260619 10:24:17.595767 2912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:39737 every 8 connection(s)
I20260619 10:24:17.596050 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:17.604285 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.596712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:41973 (local address 127.19.65.190:43483)
0619 10:24:17.596783 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.596785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.596956 (+ 171us) server_negotiation.cc:407] Connection header received
0619 10:24:17.597386 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.597387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.597397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.597452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:17.598273 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.598750 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.599738 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.599922 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.599976 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.602045 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:17.602046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:17.602046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:17.602053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:17.603193 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.603559 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.603560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.603560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.603609 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.603793 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.603794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.603794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.603887 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:17.603935 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.604149 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.604210 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:24:17.604431 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.596685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:41973)
0619 10:24:17.596831 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.596871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.597309 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.597504 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.597507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.598025 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:17.598200 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.598205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.598828 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.598829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.599668 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.599671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.599994 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.599995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.600088 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.600567 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:17.600575 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:17.601961 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:17.603268 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.603273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.603275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.603497 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.603650 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.603651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.603651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.603745 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.603943 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:17.603947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:17.604046 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.604124 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.604233 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260619 10:24:17.604923 2913 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:17.605010 2913 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:17.605136 2913 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:17.605481 2588 ts_manager.cc:194] Registered new tserver with Master: a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737)
I20260619 10:24:17.605657 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2784
I20260619 10:24:17.605765 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:24:17.605914 2588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:41973
I20260619 10:24:17.605948 19718 external_mini_cluster.cc:1428] Setting key 7f675591eb7e39f152418cfd8300c57d
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:24:17.628000 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:17.705648 2918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:17.705844 2918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:17.705868 2918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:17.707005 2918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:17.707046 2918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:17.707103 2918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:24:17.708655 2918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:17.708916 2918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:17.709134 2918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:17.714056 2924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.714041 2923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:17.714066 2926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:17.714315 2918 server_base.cc:1034] running on GCE node
I20260619 10:24:17.714547 2918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:17.714761 2918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:17.715893 2918 hybrid_clock.cc:648] HybridClock initialized: now 1781864657715868 us; error 40 us; skew 500 ppm
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:17.718187 2918 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:24:17.718542 2918 webserver.cc:460] Webserver started at http://127.19.65.131:41425/ using document root <none> and password file <none>
I20260619 10:24:17.718710 2918 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:17.718751 2918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:17.718834 2918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:24:17.719863 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "a1d76052b8be44279a5d0dba594f7b2e"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "d0ba1bee6a19c3e3e15ac366e3f0ffc1"
server_key_iv: "d7fe3b68516173df0885176fd377b3f9"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.720146 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "a1d76052b8be44279a5d0dba594f7b2e"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "d0ba1bee6a19c3e3e15ac366e3f0ffc1"
server_key_iv: "d7fe3b68516173df0885176fd377b3f9"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.721336 2918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:24:17.722025 2933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.722242 2918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:17.722312 2918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "a1d76052b8be44279a5d0dba594f7b2e"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "d0ba1bee6a19c3e3e15ac366e3f0ffc1"
server_key_iv: "d7fe3b68516173df0885176fd377b3f9"
server_key_version: "encryptionkey@0"
I20260619 10:24:17.722363 2918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:17.742246 2918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:17.744105 2918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:17.744221 2918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:17.744395 2918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:17.744757 2918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:17.744797 2918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.744846 2918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:17.744874 2918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:17.750785 2918 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45171
I20260619 10:24:17.750861 3046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45171 every 8 connection(s)
I20260619 10:24:17.751194 2918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:24:17.752043 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 2918
I20260619 10:24:17.752146 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:24:17.752334 19718 external_mini_cluster.cc:1428] Setting key fa9031c44033e9c9cb70e94cc9dad5eb
Jun 19 10:24:17 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864657, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:17.760219 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.751859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:37929 (local address 127.19.65.190:43483)
0619 10:24:17.751927 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.751929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.752108 (+ 179us) server_negotiation.cc:407] Connection header received
0619 10:24:17.752718 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.752719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.752731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.752784 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:17.753771 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.754279 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.755272 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.755498 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.755582 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.757642 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:17.757643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:17.757643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:17.757651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:17.758795 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.759218 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.759219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.759219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.759270 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.759562 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.759563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.759564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.759680 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:17.759751 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.760035 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.760120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:24:17.760411 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.751831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:37929)
0619 10:24:17.751986 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.752026 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.752600 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.752855 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.752858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.753426 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:17.753600 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.753605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.754376 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.754377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.755163 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.755165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.755592 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.755593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.755723 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.756120 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:17.756128 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:17.757534 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:17.758877 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.758884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.758886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.759103 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.759381 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:17.759382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:17.759382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:17.759491 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:17.759755 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:17.759761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:17.759890 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.759997 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.760141 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260619 10:24:17.760818 3047 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:17.760916 3047 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:17.761085 3047 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:17.761425 2588 ts_manager.cc:194] Registered new tserver with Master: a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
I20260619 10:24:17.761803 2588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:37929
I20260619 10:24:17.763854 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:24:17.770756 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.765093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.19.65.190:43483)
0619 10:24:17.765167 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.765169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.765267 (+ 98us) server_negotiation.cc:407] Connection header received
0619 10:24:17.765342 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.765343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.765353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.765408 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:17.765952 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.766404 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.767124 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.767386 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.767457 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.768550 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:17.768551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:17.768551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:17.768558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:17.769645 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.769975 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.769976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.769977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.770028 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:17.770234 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:17.770235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:17.770235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:17.770372 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:17.770474 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.770591 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.770662 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 10:24:17.772137 2588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:24:17.773648 2588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:24:17.779681 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.775947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.19.65.131:45171)
0619 10:24:17.776105 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.776110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.776120 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:24:17.776156 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.776158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.776173 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.776263 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:17.776505 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777260 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.778510 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779207 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779254 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.779346 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.779571 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.779606 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260619 10:24:17.779673 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.775781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45171 (local address 127.0.0.1:35484)
0619 10:24:17.775988 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.776017 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.776082 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.776330 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.776332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.776334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:17.776376 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.776377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777358 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.777359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.778362 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.778364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779376 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.779466 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.779475 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.779556 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.779606 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":44,"threads_started":1}
I20260619 10:24:17.780033 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.775801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.0.0.1:39496)
0619 10:24:17.776170 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.776179 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.776216 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.776625 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.776626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.776628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:17.776673 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.776675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777934 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.777935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.778876 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.778878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779805 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.779878 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.779882 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.779940 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.779974 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":21,"thread_start_us":34,"threads_started":1}
I20260619 10:24:17.780210 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.775757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39737 (local address 127.0.0.1:55564)
0619 10:24:17.775978 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.776016 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.776092 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.776563 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.776563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.776566 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:17.776607 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.776609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777915 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.777916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.779035 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.779038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779982 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.780042 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.780045 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.780095 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.780140 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":28,"thread_start_us":49,"threads_started":1}
I20260619 10:24:17.780282 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.776114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.19.65.129:40083)
0619 10:24:17.776289 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.776293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.776304 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:24:17.776343 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.776344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.776424 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.776497 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:17.776731 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777806 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.778966 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779728 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.779860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.779960 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.780006 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260619 10:24:17.780599 2981 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (DEFAULT_TABLE table=test-table [id=148fec68b47e4032a988ec7334582b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:17.780690 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.775825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.19.65.130:39737)
0619 10:24:17.776330 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.776334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.776343 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:17.776379 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.776381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.776395 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.776477 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:17.776688 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.777806 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779121 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.779838 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.779896 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.779947 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.780571 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.780615 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":85,"threads_started":1}
I20260619 10:24:17.780931 2981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:17.780882 2713 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (DEFAULT_TABLE table=test-table [id=148fec68b47e4032a988ec7334582b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:17.781155 2713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:17.783327 3061 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Bootstrap starting.
I20260619 10:24:17.783409 3062 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Bootstrap starting.
I20260619 10:24:17.784046 3062 tablet_bootstrap.cc:654] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:17.784123 3061 tablet_bootstrap.cc:654] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:17.784081 2847 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (DEFAULT_TABLE table=test-table [id=148fec68b47e4032a988ec7334582b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:17.784422 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:17.784636 3062 log.cc:824] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:17.784663 3061 log.cc:824] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:17.785354 3062 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: No bootstrap required, opened a new log
I20260619 10:24:17.785418 3061 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: No bootstrap required, opened a new log
I20260619 10:24:17.785434 3062 ts_tablet_manager.cc:1397] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:17.785531 3061 ts_tablet_manager.cc:1397] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:17.787003 3062 raft_consensus.cc:348] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.787155 3062 raft_consensus.cc:374] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:17.787197 3062 raft_consensus.cc:729] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731285bf624a4544bd96b020a65449f7, State: Initialized, Role: FOLLOWER
I20260619 10:24:17.787348 3062 consensus_queue.cc:260] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.787624 3062 ts_tablet_manager.cc:1428] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:17.787539 3061 raft_consensus.cc:348] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.787693 3061 raft_consensus.cc:374] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:17.787740 3061 raft_consensus.cc:729] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d76052b8be44279a5d0dba594f7b2e, State: Initialized, Role: FOLLOWER
I20260619 10:24:17.787753 2779 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
I20260619 10:24:17.787829 3066 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Bootstrap starting.
I20260619 10:24:17.787842 3061 consensus_queue.cc:260] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.788093 3061 ts_tablet_manager.cc:1428] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 10:24:17.788223 3047 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
I20260619 10:24:17.788771 3066 tablet_bootstrap.cc:654] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:17.789338 3066 log.cc:824] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:17.790045 3066 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: No bootstrap required, opened a new log
I20260619 10:24:17.790119 3066 ts_tablet_manager.cc:1397] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:17.791450 3066 raft_consensus.cc:348] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.791586 3066 raft_consensus.cc:374] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:17.791628 3066 raft_consensus.cc:729] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fc42d58989417b9fcc29e868c8a2cc, State: Initialized, Role: FOLLOWER
I20260619 10:24:17.791742 3066 consensus_queue.cc:260] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.791985 3066 ts_tablet_manager.cc:1428] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:24:17.792021 2913 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
W20260619 10:24:17.846611 2914 tablet.cc:2369] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:24:17.940596 2780 tablet.cc:2369] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:24:17.995672 3069 raft_consensus.cc:482] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:17.995810 3069 raft_consensus.cc:504] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:17.996140 3069 leader_election.cc:288] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171), 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:17.998940 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.996369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:49915 (local address 127.19.65.129:40083)
0619 10:24:17.996409 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.996411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.996561 (+ 150us) server_negotiation.cc:407] Connection header received
0619 10:24:17.996608 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.996609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.996620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.996673 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:17.996856 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.997337 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998242 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.998594 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998650 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.998658 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.998826 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.998869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:24:17.998986 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.996391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:60631 (local address 127.19.65.131:45171)
0619 10:24:17.996437 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:17.996439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:17.996448 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:17.996477 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:17.996478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:17.996487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:17.996558 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:17.996744 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.997262 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998246 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.998610 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998700 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.998708 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:17.998878 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 10:24:17.998920 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:24:17.998986 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.996300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45171 (local address 127.19.65.130:60631)
0619 10:24:17.996354 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.996366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.996410 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.996577 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.996578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.996580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:17.996669 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.996671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.997337 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.997338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.998132 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.998134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.998712 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.998780 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.998783 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.998856 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.998893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":36}
I20260619 10:24:17.999001 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:17.996301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.19.65.130:49915)
0619 10:24:17.996488 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:17.996498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:17.996553 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:17.996684 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:17.996684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:17.996686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:17.996790 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.996791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.997413 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.997414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.998137 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:17.998138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:17.998673 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:17.998674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:17.998757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:17.998762 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:24:17.998804 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:24:17.998852 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":14,"thread_start_us":107,"threads_started":1}
I20260619 10:24:17.999363 3001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a3bf31dd2e4071b4ecbc3340ef00ad" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d76052b8be44279a5d0dba594f7b2e" is_pre_election: true
I20260619 10:24:17.999397 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a3bf31dd2e4071b4ecbc3340ef00ad" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731285bf624a4544bd96b020a65449f7" is_pre_election: true
I20260619 10:24:17.999521 3001 raft_consensus.cc:2454] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 0.
I20260619 10:24:17.999588 2733 raft_consensus.cc:2454] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 0.
I20260619 10:24:17.999760 2801 leader_election.cc:302] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc; no voters:
I20260619 10:24:17.999943 3069 raft_consensus.cc:2781] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:24:17.999981 3069 raft_consensus.cc:482] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:24:17.999996 3069 raft_consensus.cc:3037] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:18.000667 3069 raft_consensus.cc:504] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:18.000794 3069 leader_election.cc:288] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 election: Requested vote from peers a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171), 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:18.000936 3001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a3bf31dd2e4071b4ecbc3340ef00ad" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d76052b8be44279a5d0dba594f7b2e"
I20260619 10:24:18.000955 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a3bf31dd2e4071b4ecbc3340ef00ad" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731285bf624a4544bd96b020a65449f7"
I20260619 10:24:18.001008 3001 raft_consensus.cc:3037] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:18.001034 2733 raft_consensus.cc:3037] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Advancing to term 1
W20260619 10:24:18.001729 3048 tablet.cc:2369] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:24:18.001821 2733 raft_consensus.cc:2454] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 1.
I20260619 10:24:18.001967 2802 leader_election.cc:302] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731285bf624a4544bd96b020a65449f7, a8fc42d58989417b9fcc29e868c8a2cc; no voters:
I20260619 10:24:18.001981 3001 raft_consensus.cc:2454] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 1.
I20260619 10:24:18.002048 3069 raft_consensus.cc:2781] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:24:18.002213 3069 raft_consensus.cc:686] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 1 LEADER]: Becoming Leader. State: Replica: a8fc42d58989417b9fcc29e868c8a2cc, State: Running, Role: LEADER
I20260619 10:24:18.002302 3069 consensus_queue.cc:237] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:18.003178 2587 catalog_manager.cc:5495] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130). New cstate: current_term: 1 leader_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260619 10:24:18.062880 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:18.060745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.19.65.130:39737)
0619 10:24:18.060786 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:18.060788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:18.060804 (+ 16us) server_negotiation.cc:407] Connection header received
0619 10:24:18.060856 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:18.060856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:18.060872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:18.060923 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:24:18.061028 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:18.061519 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:18.062268 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:18.062442 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:18.062491 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:18.062615 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:24:18.062703 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:24:18.062760 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:18.062792 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:24:18.062808 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:24:18.064692 2733 raft_consensus.cc:1261] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Refusing update from remote peer a8fc42d58989417b9fcc29e868c8a2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:18.064709 3001 raft_consensus.cc:1261] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Refusing update from remote peer a8fc42d58989417b9fcc29e868c8a2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:18.065022 3071 consensus_queue.cc:1035] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:18.065140 3069 consensus_queue.cc:1035] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:18.067473 3075 mvcc.cc:205] Tried to move back new op lower bound from 7298517639430250496 to 7298517639177928704. Current Snapshot: MvccSnapshot[applied={T|T < 7298517639430250496}]
I20260619 10:24:18.068523 3076 mvcc.cc:205] Tried to move back new op lower bound from 7298517639430250496 to 7298517639177928704. Current Snapshot: MvccSnapshot[applied={T|T < 7298517639430250496 or (T in {7298517639430250496})}]
I20260619 10:24:18.069526 2588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:18.069604 2588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:18.070271 2588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Sending DeleteTablet for 3 replicas of tablet 21a3bf31dd2e4071b4ecbc3340ef00ad
I20260619 10:24:18.070573 2847 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55564
I20260619 10:24:18.070623 2981 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35484
I20260619 10:24:18.070798 3084 tablet_replica.cc:330] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: stopping tablet replica
I20260619 10:24:18.070807 3085 tablet_replica.cc:330] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: stopping tablet replica
I20260619 10:24:18.070827 2713 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:39496
I20260619 10:24:18.070895 3085 raft_consensus.cc:2229] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:24:18.070953 3084 raft_consensus.cc:2229] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:24:18.070971 3085 raft_consensus.cc:2258] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:18.071072 3084 raft_consensus.cc:2258] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:18.071127 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2650
I20260619 10:24:18.071882 3085 ts_tablet_manager.cc:1905] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:18.072031 3084 ts_tablet_manager.cc:1905] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:18.073716 3084 ts_tablet_manager.cc:1918] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:18.073767 3085 ts_tablet_manager.cc:1918] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:18.073788 3084 log.cc:1196] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/21a3bf31dd2e4071b4ecbc3340ef00ad
I20260619 10:24:18.073830 3085 log.cc:1196] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/21a3bf31dd2e4071b4ecbc3340ef00ad
I20260619 10:24:18.074095 3084 ts_tablet_manager.cc:1939] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a1d76052b8be44279a5d0dba594f7b2e: Deleting consensus metadata
I20260619 10:24:18.074096 3085 ts_tablet_manager.cc:1939] T 21a3bf31dd2e4071b4ecbc3340ef00ad P a8fc42d58989417b9fcc29e868c8a2cc: Deleting consensus metadata
I20260619 10:24:18.074484 2573 catalog_manager.cc:4841] TS a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (table test-table [id=148fec68b47e4032a988ec7334582b64]) successfully deleted
I20260619 10:24:18.074549 2575 catalog_manager.cc:4841] TS a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (table test-table [id=148fec68b47e4032a988ec7334582b64]) successfully deleted
W20260619 10:24:18.077459 2574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40083 (error 108)
I20260619 10:24:18.077642 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2784
I20260619 10:24:18.078688 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:18.078557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.0.0.1:39498)
0619 10:24:18.078614 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:18.078664 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 10:24:18.078833 2574 catalog_manager.cc:4584] TS 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40083: connect: Connection refused (error 111)
I20260619 10:24:18.082475 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2918
I20260619 10:24:18.087071 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 2556
I20260619 10:24:38.092017 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--webserver_interface=127.19.65.190
--webserver_port=43409
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:38.172578 3088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:38.172780 3088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:38.172803 3088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:38.173991 3088 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:24:38.174037 3088 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:24:38.174053 3088 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:24:38.174108 3088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:24:38.174129 3088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:38.174157 3088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:38.174196 3088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:24:38.174240 3088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:24:38.175770 3088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=43409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:38.176026 3088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:38.176318 3088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:38.185137 3094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:38.185123 3095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.185159 3088 server_base.cc:1034] running on GCE node
W20260619 10:24:38.185123 3097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.185675 3088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:38.186026 3088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:38.187196 3088 hybrid_clock.cc:648] HybridClock initialized: now 1781864678187185 us; error 30 us; skew 500 ppm
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:38.189764 3088 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:24:38.190189 3088 webserver.cc:460] Webserver started at http://127.19.65.190:43409/ using document root <none> and password file <none>
I20260619 10:24:38.190433 3088 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:38.190488 3088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:38.191908 3088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:24:38.192592 3104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.192772 3088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:24:38.192859 3088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "230de586ac9b423c9834e331c1f1dfc8"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "607bd41616e3fe7bbb5e537b12cabd18"
server_key_iv: "e3fa24c749cec89ac193d5ceaf459226"
server_key_version: "encryptionkey@0"
I20260619 10:24:38.193244 3088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:38.201171 3088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:38.202925 3088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:38.203071 3088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:38.207520 3088 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43483
I20260619 10:24:38.207600 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:43483 every 8 connection(s)
I20260619 10:24:38.207929 3088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:24:38.209378 3157 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:24:38.210063 3157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap starting.
I20260619 10:24:38.216670 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3088
I20260619 10:24:38.216943 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:40083
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=38637
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:38.218248 3157 log.cc:824] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:38.220305 3157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:38.220593 3157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap complete.
I20260619 10:24:38.222818 3157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:38.223088 3157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Initialized, Role: FOLLOWER
I20260619 10:24:38.223276 3157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:38.223397 3157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:24:38.223454 3157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:24:38.223500 3157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:24:38.224427 3157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:38.224540 3157 leader_election.cc:302] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230de586ac9b423c9834e331c1f1dfc8; no voters:
I20260619 10:24:38.224730 3157 leader_election.cc:288] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:24:38.224771 3161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:24:38.224880 3161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 LEADER]: Becoming Leader. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Running, Role: LEADER
I20260619 10:24:38.224980 3157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:24:38.225015 3161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:38.225397 3163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230de586ac9b423c9834e331c1f1dfc8. Latest consensus state: current_term: 2 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:38.225410 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:38.225515 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:38.225502 3163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:38.226151 3170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:24:38.227042 3170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:24:38.227138 3170 catalog_manager.cc:1259] Loaded cluster ID: 334fa5febb664ae4b251fc2fbdd85df9
I20260619 10:24:38.227169 3170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:24:38.228379 3170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:24:38.228541 3170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Loaded TSK: 0
I20260619 10:24:38.238338 3170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Generated new TSK 1
I20260619 10:24:38.238417 3170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:24:38.300326 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:38.300511 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:38.300534 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:38.301647 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:38.301685 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:38.301712 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:24:38.303114 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=38637
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:38.303375 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:38.303611 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:38.309268 3184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:38.309309 3185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.309499 3159 server_base.cc:1034] running on GCE node
W20260619 10:24:38.309623 3187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.309854 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:38.310119 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:38.311276 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781864678311259 us; error 33 us; skew 500 ppm
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:38.313949 3159 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:24:38.314407 3159 webserver.cc:460] Webserver started at http://127.19.65.129:38637/ using document root <none> and password file <none>
I20260619 10:24:38.314643 3159 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:38.314725 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:38.316100 3159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:24:38.316782 3194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.316951 3159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.317044 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "731285bf624a4544bd96b020a65449f7"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "ade113217354518cbecdb5b1a69bf5f7"
server_key_iv: "9d9e68d8504955d78678d6aaae55fbf3"
server_key_version: "encryptionkey@0"
I20260619 10:24:38.317370 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:38.337668 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:38.339951 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:38.340116 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:38.340339 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:38.340765 3201 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:24:38.342542 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:24:38.342603 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:24:38.342639 3159 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:24:38.343255 3159 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:24:38.343326 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.343370 3201 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Bootstrap starting.
I20260619 10:24:38.351197 3159 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40083
I20260619 10:24:38.351516 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40083 every 8 connection(s)
I20260619 10:24:38.351696 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:24:38.352144 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3159
I20260619 10:24:38.352416 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:39737
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=37335
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:38.353499 3201 log.cc:824] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:38.356216 3201 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:38.356540 3201 tablet_bootstrap.cc:492] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Bootstrap complete.
I20260619 10:24:38.357692 3201 ts_tablet_manager.cc:1397] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260619 10:24:38.358758 3201 raft_consensus.cc:348] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.359018 3201 raft_consensus.cc:729] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731285bf624a4544bd96b020a65449f7, State: Initialized, Role: FOLLOWER
I20260619 10:24:38.359182 3201 consensus_queue.cc:260] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.359486 3201 ts_tablet_manager.cc:1428] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:38.365911 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.353919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:40235 (local address 127.19.65.190:43483)
0619 10:24:38.354087 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.354096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.354312 (+ 216us) server_negotiation.cc:407] Connection header received
0619 10:24:38.355109 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.355116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.355169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.355312 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.356138 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.357557 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.359223 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.359782 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.359870 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.362310 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.362314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.362315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.362326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.363984 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.364589 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.364591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.364592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.364689 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.364946 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.364947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.364947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.365173 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.365277 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.365613 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.365730 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260619 10:24:38.365931 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.353859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:40235)
0619 10:24:38.354148 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.354191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.354844 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.355371 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.355375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.355704 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:38.355978 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.355984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.357675 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.357676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.359138 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.359141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.359871 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.359873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.360029 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.360571 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:38.360582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:38.362184 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:38.364140 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.364148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.364150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.364490 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.364765 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.364765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.364766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.364872 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.365270 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:38.365276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:38.365437 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.365567 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.365730 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260619 10:24:38.366606 3309 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:38.366704 3309 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:38.366864 3309 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:38.367616 3121 ts_manager.cc:194] Registered new tserver with Master: 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:38.368667 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:40235
I20260619 10:24:38.369617 3309 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.129@KRBTEST.COM
I20260619 10:24:38.376358 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.368278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.19.65.129:40083)
0619 10:24:38.368412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.368416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.368619 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:24:38.368710 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.368712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.368727 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.368803 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.369239 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.370216 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.371356 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.371643 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.371703 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.373685 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.373692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.373692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.373698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.375156 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.375547 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.375549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.375549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.375602 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.375805 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.375806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.375806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.375934 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.376018 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.376193 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.376275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260619 10:24:38.376358 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.368206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.0.0.1:43268)
0619 10:24:38.368494 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.368535 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.368610 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.368811 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.368816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.369112 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:38.369159 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.369161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.370308 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.370309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.371207 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.371209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.371716 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.371717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.371791 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.372211 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:38.372216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:38.373576 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:38.375253 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.375259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.375261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.375463 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.375665 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.375666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.375666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.375719 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.376011 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:38.376016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:38.376063 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.376159 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.376266 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260619 10:24:38.376880 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a3bf31dd2e4071b4ecbc3340ef00ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43268
I20260619 10:24:38.377314 3318 tablet_replica.cc:330] stopping tablet replica
I20260619 10:24:38.377408 3318 raft_consensus.cc:2229] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:24:38.377466 3318 raft_consensus.cc:2258] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:38.377733 3318 ts_tablet_manager.cc:1905] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:38.380088 3318 ts_tablet_manager.cc:1918] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:38.380160 3318 log.cc:1196] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal/wals/21a3bf31dd2e4071b4ecbc3340ef00ad
I20260619 10:24:38.380371 3318 ts_tablet_manager.cc:1939] T 21a3bf31dd2e4071b4ecbc3340ef00ad P 731285bf624a4544bd96b020a65449f7: Deleting consensus metadata
I20260619 10:24:38.380860 3107 catalog_manager.cc:4841] TS 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083): tablet 21a3bf31dd2e4071b4ecbc3340ef00ad (table test-table [id=148fec68b47e4032a988ec7334582b64]) successfully deleted
W20260619 10:24:38.435109 3311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:38.435312 3311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:38.435336 3311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:38.436429 3311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:38.436462 3311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:38.436487 3311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:24:38.437875 3311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:39737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=37335
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:38.438043 3311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:38.438202 3311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:38.443050 3326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:38.443063 3324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:38.443145 3323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.443410 3311 server_base.cc:1034] running on GCE node
I20260619 10:24:38.443617 3311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:38.443853 3311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:38.445011 3311 hybrid_clock.cc:648] HybridClock initialized: now 1781864678444991 us; error 49 us; skew 500 ppm
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:38.447417 3311 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:24:38.447867 3311 webserver.cc:460] Webserver started at http://127.19.65.130:37335/ using document root <none> and password file <none>
I20260619 10:24:38.448081 3311 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:38.448155 3311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:38.449427 3311 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.449980 3333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.450158 3311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.450251 3311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "a8fc42d58989417b9fcc29e868c8a2cc"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "554d7fbbc15413db786ba6d7a92aef57"
server_key_iv: "0a93819b94248effcd19f256b037fe3e"
server_key_version: "encryptionkey@0"
I20260619 10:24:38.450577 3311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:38.466470 3311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:38.468250 3311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:38.468400 3311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:38.468617 3311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:38.468884 3311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:38.468935 3311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.468971 3311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:38.469002 3311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.474524 3311 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:39737
I20260619 10:24:38.474572 3446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:39737 every 8 connection(s)
I20260619 10:24:38.474866 3311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:24:38.478132 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3311
I20260619 10:24:38.478392 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:45171
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=41425
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:38.484885 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.475528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:37203)
0619 10:24:38.475687 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.475724 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.476193 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.476397 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.476400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.477015 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:38.477201 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.477206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.477836 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.477837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.478766 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.478769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.479282 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.479283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.479464 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.479989 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:38.479999 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:38.481498 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:38.483402 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.483408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.483409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.483644 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.483887 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.483888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.483889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.484009 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.484348 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:38.484353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:38.484458 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.484617 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.484738 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260619 10:24:38.484911 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.475544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:37203 (local address 127.19.65.190:43483)
0619 10:24:38.475587 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.475589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.475808 (+ 219us) server_negotiation.cc:407] Connection header received
0619 10:24:38.476269 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.476270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.476282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.476337 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.477288 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.477761 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.478849 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.479181 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.479252 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.481600 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.481601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.481602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.481610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.483288 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.483718 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.483720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.483720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.483788 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.484104 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.484105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.484105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.484260 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.484349 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.484672 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.484777 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:24:38.485430 3447 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:38.485549 3447 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:38.485743 3447 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:38.486197 3121 ts_manager.cc:194] Registered new tserver with Master: a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737)
I20260619 10:24:38.486747 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:37203
W20260619 10:24:38.558895 3450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:38.559082 3450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:38.559105 3450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:38.560261 3450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:38.560302 3450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:38.560328 3450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:24:38.562359 3450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:45171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=41425
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:38.562536 3450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:38.562703 3450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:38.567760 3458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:38.567806 3455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.567932 3450 server_base.cc:1034] running on GCE node
W20260619 10:24:38.567761 3456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:38.568207 3450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:38.568442 3450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:38.569633 3450 hybrid_clock.cc:648] HybridClock initialized: now 1781864678569619 us; error 29 us; skew 500 ppm
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:38.572296 3450 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:24:38.572721 3450 webserver.cc:460] Webserver started at http://127.19.65.131:41425/ using document root <none> and password file <none>
I20260619 10:24:38.572937 3450 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:38.573040 3450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:38.574344 3450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.574914 3465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.575105 3450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:38.575201 3450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "a1d76052b8be44279a5d0dba594f7b2e"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "d0ba1bee6a19c3e3e15ac366e3f0ffc1"
server_key_iv: "d7fe3b68516173df0885176fd377b3f9"
server_key_version: "encryptionkey@0"
I20260619 10:24:38.575580 3450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:38.594298 3450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:38.596619 3450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:38.596782 3450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:38.597002 3450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:38.597291 3450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:38.597345 3450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.597384 3450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:38.597409 3450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:38.602952 3450 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45171
I20260619 10:24:38.603020 3578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45171 every 8 connection(s)
I20260619 10:24:38.603407 3450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:24:38 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864678, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:24:38.611663 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.604059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:49875 (local address 127.19.65.190:43483)
0619 10:24:38.604105 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.604107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.604311 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:24:38.604759 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.604760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.604769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.604818 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.605660 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.606120 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.607022 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.607222 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.607272 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.609157 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.609158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.609159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.609164 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.610544 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.610888 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.610889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.610889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.610935 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.611129 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.611129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.611129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.611219 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.611264 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.611532 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.611604 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:24:38.611781 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.604024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:49875)
0619 10:24:38.604179 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.604216 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.604674 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.604873 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.604876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.605413 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:24:38.605581 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.605585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.606195 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.606196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.606944 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.606946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.607307 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.607308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.607401 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.607759 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:24:38.607767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:24:38.609077 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:24:38.610624 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.610628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.610629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.610814 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.610989 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:24:38.610990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:24:38.610990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:24:38.611078 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:24:38.611270 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:24:38.611273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:24:38.611425 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.611504 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.611612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260619 10:24:38.612213 3579 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43483
I20260619 10:24:38.612306 3579 heartbeater.cc:464] Registering TS with master...
I20260619 10:24:38.612458 3579 heartbeater.cc:511] Master 127.19.65.190:43483 requested a full tablet report, sending...
I20260619 10:24:38.612792 3121 ts_manager.cc:194] Registered new tserver with Master: a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
I20260619 10:24:38.613134 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:49875
I20260619 10:24:38.613602 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3450
I20260619 10:24:38.619982 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.614012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.19.65.190:43483)
0619 10:24:38.614070 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.614072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.614205 (+ 133us) server_negotiation.cc:407] Connection header received
0619 10:24:38.614278 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.614279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.614288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.614342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.614895 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.615460 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.616199 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.616394 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.616465 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.617475 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.617476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.617477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.617483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.618788 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.619180 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.619181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.619181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.619233 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.619476 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.619477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.619477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.619582 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.619780 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.619819 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.619891 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:24:38.620708 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:24:38.623328 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.621097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.19.65.190:43483)
0619 10:24:38.621145 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.621147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.621169 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:24:38.621233 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.621233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.621266 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.621346 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:24:38.621446 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.621867 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.622602 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.622825 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.622897 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.623033 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:24:38.623072 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 10:24:38.623248 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59844: token expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:24:38.623375 3314 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59844: token expired
W20260619 10:24:38.623483 3053 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.65.190:43483)
I20260619 10:24:38.629226 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.623708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.19.65.190:43483)
0619 10:24:38.623760 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.623762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.623771 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:38.623811 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.623812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.623820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.623882 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:24:38.624311 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.624807 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.625516 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.625745 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.625821 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.626933 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:24:38.626934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:24:38.626934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:24:38.626940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:24:38.628164 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.628495 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.628496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.628496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.628562 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:24:38.628767 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:24:38.628768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:24:38.628768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:24:38.628872 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:24:38.628965 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.629067 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.629138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:24:38.630227 3121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:24:38.631896 3121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:24:38.634171 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 (DEFAULT_TABLE table=test-table [id=f43548f21a364e15a3233e8e2bc277fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:38.635210 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54623d6fc450e9952f45c5bc30d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:38.636605 3201 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Bootstrap starting.
I20260619 10:24:38.636937 3201 tablet_bootstrap.cc:654] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:38.637485 3201 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: No bootstrap required, opened a new log
I20260619 10:24:38.637539 3201 ts_tablet_manager.cc:1397] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.637892 3201 raft_consensus.cc:348] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.637971 3201 raft_consensus.cc:374] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:38.638011 3201 raft_consensus.cc:729] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731285bf624a4544bd96b020a65449f7, State: Initialized, Role: FOLLOWER
I20260619 10:24:38.638054 3201 consensus_queue.cc:260] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.638159 3201 ts_tablet_manager.cc:1428] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:38.638183 3584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.633851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45171 (local address 127.0.0.1:38142)
0619 10:24:38.634308 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.634319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.634380 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.634580 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.634581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.634583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:38.634621 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.634622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.635900 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.635901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.637032 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.637035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.637946 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.637946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.637999 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.638040 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.638086 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.638120 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":18,"thread_start_us":87,"threads_started":1}
I20260619 10:24:38.638609 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.634065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.19.65.131:45171)
0619 10:24:38.634212 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.634217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.634369 (+ 152us) server_negotiation.cc:407] Connection header received
0619 10:24:38.634443 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.634445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.634459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.634560 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:38.634688 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.635813 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.637110 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.637844 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.637942 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.637999 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.638456 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.638505 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260619 10:24:38.639356 3513 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 (DEFAULT_TABLE table=test-table [id=f43548f21a364e15a3233e8e2bc277fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:38.639658 3513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54623d6fc450e9952f45c5bc30d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:38.640110 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.633881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.19.65.130:39737)
0619 10:24:38.636707 (+ 2826us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.636710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.636720 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:24:38.636755 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.636757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.636771 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.636848 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:38.636972 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.637812 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.639025 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.639813 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.639877 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.639912 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.640028 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.640054 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2790,"thread_start_us":94,"threads_started":1}
I20260619 10:24:38.640141 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.633716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39737 (local address 127.0.0.1:34580)
0619 10:24:38.633814 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.633825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.633855 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.636860 (+ 3005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.636861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.636863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:38.636903 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.636904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.637912 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.637913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.638954 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.638957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.639882 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.639883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.639948 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.639952 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.640020 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.640068 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 10:24:38.640874 3381 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 (DEFAULT_TABLE table=test-table [id=f43548f21a364e15a3233e8e2bc277fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:24:38.641191 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54623d6fc450e9952f45c5bc30d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:24:38.642657 3588 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Bootstrap starting.
I20260619 10:24:38.643375 3588 tablet_bootstrap.cc:654] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:38.643965 3588 log.cc:824] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:38.644241 3589 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Bootstrap starting.
I20260619 10:24:38.644642 3588 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: No bootstrap required, opened a new log
I20260619 10:24:38.644726 3588 ts_tablet_manager.cc:1397] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:24:38.644795 3589 tablet_bootstrap.cc:654] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Neither blocks nor log segments found. Creating new log.
I20260619 10:24:38.645242 3589 log.cc:824] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:38.645905 3589 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: No bootstrap required, opened a new log
I20260619 10:24:38.645961 3589 ts_tablet_manager.cc:1397] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:24:38.646351 3588 raft_consensus.cc:348] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.646502 3588 raft_consensus.cc:374] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:38.646546 3588 raft_consensus.cc:729] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d76052b8be44279a5d0dba594f7b2e, State: Initialized, Role: FOLLOWER
I20260619 10:24:38.646667 3588 consensus_queue.cc:260] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.646955 3579 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
I20260619 10:24:38.647195 3588 ts_tablet_manager.cc:1428] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 10:24:38.647951 3589 raft_consensus.cc:348] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.648075 3589 raft_consensus.cc:374] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:24:38.648104 3589 raft_consensus.cc:729] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fc42d58989417b9fcc29e868c8a2cc, State: Initialized, Role: FOLLOWER
I20260619 10:24:38.648211 3589 consensus_queue.cc:260] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.648458 3589 ts_tablet_manager.cc:1428] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:24:38.648473 3447 heartbeater.cc:503] Master 127.19.65.190:43483 was elected leader, sending a full tablet report...
I20260619 10:24:38.684777 3593 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:24:38.684890 3593 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.685286 3593 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171), 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:38.688244 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.685460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45171 (local address 127.19.65.130:59297)
0619 10:24:38.685522 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.685535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.685580 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.685783 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.685784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.685785 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:38.685860 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.685862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.686478 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.686479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.687278 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.687280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.688066 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.688066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.688122 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.688125 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.688167 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.688197 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 10:24:38.688292 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.685608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59297 (local address 127.19.65.131:45171)
0619 10:24:38.685659 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.685661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.685671 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:24:38.685698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.685699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.685709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.685768 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:38.685938 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.686410 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.687439 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.687989 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.688063 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.688071 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.688193 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.688228 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 10:24:38.688534 3533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54623d6fc450e9952f45c5bc30d61" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d76052b8be44279a5d0dba594f7b2e" is_pre_election: true
I20260619 10:24:38.688711 3533 raft_consensus.cc:2454] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 0.
I20260619 10:24:38.688722 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:50933 (local address 127.19.65.129:40083)
0619 10:24:38.685608 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.685610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.685737 (+ 127us) server_negotiation.cc:407] Connection header received
0619 10:24:38.685785 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.685786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.685813 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.685885 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:24:38.686063 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.686566 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.687708 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.688327 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.688397 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.688474 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.688622 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.688667 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:24:38.688728 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.685463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.19.65.130:50933)
0619 10:24:38.685659 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.685669 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:38.685733 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:38.685923 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:24:38.685924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:24:38.685925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:24:38.685988 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.685989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.686645 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.686646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.687622 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:24:38.687624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.688473 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.688474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:24:38.688539 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.688544 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:24:38.688605 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:24:38.688634 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260619 10:24:38.688920 3335 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc; no voters:
I20260619 10:24:38.689081 3593 raft_consensus.cc:2781] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:24:38.689092 3263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54623d6fc450e9952f45c5bc30d61" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731285bf624a4544bd96b020a65449f7" is_pre_election: true
I20260619 10:24:38.689163 3593 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:24:38.689195 3593 raft_consensus.cc:3037] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:38.689232 3263 raft_consensus.cc:2454] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 0.
I20260619 10:24:38.689975 3593 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.690114 3593 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 election: Requested vote from peers a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171), 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083)
I20260619 10:24:38.690251 3263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54623d6fc450e9952f45c5bc30d61" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731285bf624a4544bd96b020a65449f7"
I20260619 10:24:38.690279 3533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54623d6fc450e9952f45c5bc30d61" candidate_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d76052b8be44279a5d0dba594f7b2e"
I20260619 10:24:38.690348 3263 raft_consensus.cc:3037] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:38.690348 3533 raft_consensus.cc:3037] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:24:38.690980 3263 raft_consensus.cc:2454] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 1.
I20260619 10:24:38.691104 3533 raft_consensus.cc:2454] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fc42d58989417b9fcc29e868c8a2cc in term 1.
I20260619 10:24:38.691277 3336 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731285bf624a4544bd96b020a65449f7, a8fc42d58989417b9fcc29e868c8a2cc; no voters:
I20260619 10:24:38.691459 3593 raft_consensus.cc:2781] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:24:38.691630 3593 raft_consensus.cc:686] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 1 LEADER]: Becoming Leader. State: Replica: a8fc42d58989417b9fcc29e868c8a2cc, State: Running, Role: LEADER
I20260619 10:24:38.691728 3593 consensus_queue.cc:237] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:38.692536 3121 catalog_manager.cc:5495] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130). New cstate: current_term: 1 leader_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260619 10:24:38.711447 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.709324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.19.65.130:39737)
0619 10:24:38.709363 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:24:38.709365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:24:38.709374 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:24:38.709451 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:24:38.709451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:24:38.709465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:24:38.709521 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:24:38.709656 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.710092 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.710797 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:24:38.710985 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:24:38.711028 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:24:38.711174 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:24:38.711253 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:24:38.711356 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 10:24:38.711383 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 10:24:38.711403 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:24:38.713008 3263 raft_consensus.cc:1261] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Refusing update from remote peer a8fc42d58989417b9fcc29e868c8a2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:38.713024 3533 raft_consensus.cc:1261] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Refusing update from remote peer a8fc42d58989417b9fcc29e868c8a2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:24:38.713327 3593 consensus_queue.cc:1035] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:38.713497 3595 consensus_queue.cc:1035] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:24:38.715708 3597 mvcc.cc:205] Tried to move back new op lower bound from 7298517724006035456 to 7298517723921838080. Current Snapshot: MvccSnapshot[applied={T|T < 7298517724006035456 or (T in {7298517724006035456})}]
I20260619 10:24:38.715857 3598 mvcc.cc:205] Tried to move back new op lower bound from 7298517724006035456 to 7298517723921838080. Current Snapshot: MvccSnapshot[applied={T|T < 7298517724006035456}]
I20260619 10:24:38.716408 3600 mvcc.cc:205] Tried to move back new op lower bound from 7298517724006035456 to 7298517723921838080. Current Snapshot: MvccSnapshot[applied={T|T < 7298517724006035456 or (T in {7298517724006035456})}]
I20260619 10:24:38.717509 3121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:38.717604 3121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:24:38.718308 3121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Sending DeleteTablet for 3 replicas of tablet 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:24:38.718631 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43268
I20260619 10:24:38.718642 3513 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:38142
I20260619 10:24:38.718720 3318 tablet_replica.cc:330] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: stopping tablet replica
I20260619 10:24:38.718763 3318 raft_consensus.cc:2229] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:24:38.718813 3318 raft_consensus.cc:2258] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:38.718852 3606 tablet_replica.cc:330] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: stopping tablet replica
I20260619 10:24:38.718930 3606 raft_consensus.cc:2229] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:24:38.718946 3381 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f54623d6fc450e9952f45c5bc30d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:24:38 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:34580
I20260619 10:24:38.718986 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3159
I20260619 10:24:38.719039 3606 raft_consensus.cc:2258] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:38.719545 3607 tablet_replica.cc:330] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: stopping tablet replica
I20260619 10:24:38.719630 3607 raft_consensus.cc:2229] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:24:38.719712 3607 raft_consensus.cc:2258] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:24:38.719964 3606 ts_tablet_manager.cc:1905] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:38.720114 3607 ts_tablet_manager.cc:1905] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:24:38.721829 3607 ts_tablet_manager.cc:1918] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:38.721887 3607 log.cc:1196] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/43f54623d6fc450e9952f45c5bc30d61
I20260619 10:24:38.721876 3606 ts_tablet_manager.cc:1918] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:24:38.721946 3606 log.cc:1196] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/43f54623d6fc450e9952f45c5bc30d61
I20260619 10:24:38.722191 3607 ts_tablet_manager.cc:1939] T 43f54623d6fc450e9952f45c5bc30d61 P a8fc42d58989417b9fcc29e868c8a2cc: Deleting consensus metadata
I20260619 10:24:38.722236 3606 ts_tablet_manager.cc:1939] T 43f54623d6fc450e9952f45c5bc30d61 P a1d76052b8be44279a5d0dba594f7b2e: Deleting consensus metadata
I20260619 10:24:38.722555 3108 catalog_manager.cc:4841] TS a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): tablet 43f54623d6fc450e9952f45c5bc30d61 (table test-table [id=f43548f21a364e15a3233e8e2bc277fb]) successfully deleted
I20260619 10:24:38.722635 3106 catalog_manager.cc:4841] TS a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): tablet 43f54623d6fc450e9952f45c5bc30d61 (table test-table [id=f43548f21a364e15a3233e8e2bc277fb]) successfully deleted
W20260619 10:24:38.725706 3107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40083 (error 108)
I20260619 10:24:38.725975 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3311
I20260619 10:24:38.728199 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:38.728047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40083 (local address 127.0.0.1:43278)
0619 10:24:38.728102 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:38.728137 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 10:24:38.728336 3107 catalog_manager.cc:4584] TS 731285bf624a4544bd96b020a65449f7 (127.19.65.129:40083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43f54623d6fc450e9952f45c5bc30d61: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40083: connect: Connection refused (error 111)
I20260619 10:24:38.731278 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3450
I20260619 10:24:38.735505 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3088
I20260619 10:24:58.740589 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--webserver_interface=127.19.65.190
--webserver_port=43409
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:58.820217 3609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:58.820451 3609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:58.820488 3609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:58.821591 3609 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:24:58.821659 3609 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:24:58.821681 3609 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:24:58.821699 3609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:24:58.821732 3609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:58.821756 3609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:58.821782 3609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:24:58.821806 3609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:24:58.823235 3609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=43409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:58.823511 3609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:58.823731 3609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:24:58.828907 3609 server_base.cc:1034] running on GCE node
W20260619 10:24:58.829015 3615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:58.828981 3617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:58.829025 3614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:58.829454 3609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:58.829747 3609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:58.830910 3609 hybrid_clock.cc:648] HybridClock initialized: now 1781864698830898 us; error 31 us; skew 500 ppm
Jun 19 10:24:58 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:58.834408 3609 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:24:58.835008 3609 webserver.cc:460] Webserver started at http://127.19.65.190:43409/ using document root <none> and password file <none>
I20260619 10:24:58.835254 3609 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:58.835335 3609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:58.836885 3609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:58.837740 3624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:58.838531 3609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:58.838640 3609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "230de586ac9b423c9834e331c1f1dfc8"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "607bd41616e3fe7bbb5e537b12cabd18"
server_key_iv: "e3fa24c749cec89ac193d5ceaf459226"
server_key_version: "encryptionkey@0"
I20260619 10:24:58.838987 3609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:58.861704 3609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:58.863473 3609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:58.863639 3609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:58.867614 3609 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43483
I20260619 10:24:58.867650 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:43483 every 8 connection(s)
I20260619 10:24:58.868660 3609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:24:58.870039 3677 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:24:58.870677 3677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap starting.
I20260619 10:24:58.875411 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3609
I20260619 10:24:58.875674 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:40083
--local_ip_for_outbound_sockets=127.19.65.129
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=38637
--webserver_interface=127.19.65.129
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:58.879217 3677 log.cc:824] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:58.881314 3677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:58.881608 3677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Bootstrap complete.
I20260619 10:24:58.883872 3677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:58.884109 3677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Initialized, Role: FOLLOWER
I20260619 10:24:58.884306 3677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:58.884402 3677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:24:58.884454 3677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:24:58.884501 3677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 2 FOLLOWER]: Advancing to term 3
I20260619 10:24:58.885411 3677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:58.885527 3677 leader_election.cc:302] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230de586ac9b423c9834e331c1f1dfc8; no voters:
I20260619 10:24:58.885717 3677 leader_election.cc:288] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 10:24:58.885751 3681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 10:24:58.885866 3681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [term 3 LEADER]: Becoming Leader. State: Replica: 230de586ac9b423c9834e331c1f1dfc8, State: Running, Role: LEADER
I20260619 10:24:58.885967 3677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:24:58.885948 3681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } }
I20260619 10:24:58.886251 3681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230de586ac9b423c9834e331c1f1dfc8. Latest consensus state: current_term: 3 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:58.886318 3681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:58.886462 3682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "230de586ac9b423c9834e331c1f1dfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230de586ac9b423c9834e331c1f1dfc8" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43483 } } }
I20260619 10:24:58.886530 3686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:24:58.886584 3682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:24:58.887336 3686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:24:58.887439 3686 catalog_manager.cc:1259] Loaded cluster ID: 334fa5febb664ae4b251fc2fbdd85df9
I20260619 10:24:58.887475 3686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:24:58.888566 3686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:24:58.888736 3686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Loaded TSK: 0
I20260619 10:24:58.888779 3686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 230de586ac9b423c9834e331c1f1dfc8: Loaded TSK: 1
F20260619 10:24:58.889019 3686 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faaee04e9ad google::LogMessage::Fail() at ??:0
@ 0x7faaee05188a google::LogMessage::SendToLog() at ??:0
@ 0x7faaee04e4de google::LogMessage::Flush() at ??:0
@ 0x7faaee04fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faaf07bb104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faaf07b8d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faaf5fad1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faaf5fad48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faaf5fad594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faaf5f73c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faaf5f7dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faaeff1c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faaeff11954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faaf0d756db start_thread at ??:0
@ 0x7faaecb4c61f clone at ??:0
W20260619 10:24:58.966430 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:58.966647 3679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:58.966688 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:58.967890 3679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:58.967947 3679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:58.967993 3679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:24:58.969384 3679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=38637
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:58.969635 3679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:58.969844 3679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:58.974651 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:58.974651 3718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:58.974831 3679 server_base.cc:1034] running on GCE node
W20260619 10:24:58.974689 3719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:58.975119 3679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:58.975435 3679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:58.976588 3679 hybrid_clock.cc:648] HybridClock initialized: now 1781864698976579 us; error 27 us; skew 500 ppm
Jun 19 10:24:58 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:58.979010 3679 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:24:58.979540 3679 webserver.cc:460] Webserver started at http://127.19.65.129:38637/ using document root <none> and password file <none>
I20260619 10:24:58.979753 3679 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:58.979815 3679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:58.981168 3679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:58.981837 3728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:58.982024 3679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:58.982089 3679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "731285bf624a4544bd96b020a65449f7"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "ade113217354518cbecdb5b1a69bf5f7"
server_key_iv: "9d9e68d8504955d78678d6aaae55fbf3"
server_key_version: "encryptionkey@0"
I20260619 10:24:58.982406 3679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:59.007920 3679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:59.010025 3679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:59.010164 3679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:59.010402 3679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:59.010864 3735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:24:59.012880 3679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:24:59.012941 3679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:24:59.012988 3679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:24:59.013597 3679 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:24:59.013665 3679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:24:59.013733 3735 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Bootstrap starting.
I20260619 10:24:59.021036 3679 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40083
I20260619 10:24:59.021436 3679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:24:59.022012 3842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40083 every 8 connection(s)
I20260619 10:24:59.025341 3735 log.cc:824] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:24:59.027964 3735 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:24:59.028306 3735 tablet_bootstrap.cc:492] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Bootstrap complete.
I20260619 10:24:59.029542 3735 ts_tablet_manager.cc:1397] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260619 10:24:59.030925 3735 raft_consensus.cc:348] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:59.031258 3735 raft_consensus.cc:729] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731285bf624a4544bd96b020a65449f7, State: Initialized, Role: FOLLOWER
I20260619 10:24:59.031342 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3679
I20260619 10:24:59.031443 3735 consensus_queue.cc:260] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:24:59.031648 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:39737
--local_ip_for_outbound_sockets=127.19.65.130
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=37335
--webserver_interface=127.19.65.130
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:24:59.031726 3735 ts_tablet_manager.cc:1428] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260619 10:24:59.110008 3845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:24:59.022163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:34297)
0619 10:24:59.022347 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.022382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:24:59.022828 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:24:59.109785 (+ 86957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
W20260619 10:24:59.110214 3843 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:43483 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:43483: Client connection negotiation failed: client connection to 127.19.65.190:43483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:24:59.110539 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.110419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:39505)
0619 10:24:59.110472 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.110519 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:24:59.110958 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.110844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:41521)
0619 10:24:59.110908 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.110938 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 10:24:59.111114 3843 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:24:59.132257 3848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:59.132433 3848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:59.132455 3848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:59.133580 3848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:59.133635 3848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:59.133723 3848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:24:59.135131 3848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:39737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=37335
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:59.135366 3848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:59.135588 3848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:59.140130 3856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:59.140298 3854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:59.140300 3853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:59.140518 3848 server_base.cc:1034] running on GCE node
I20260619 10:24:59.140699 3848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:59.140882 3848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:59.142041 3848 hybrid_clock.cc:648] HybridClock initialized: now 1781864699142023 us; error 30 us; skew 500 ppm
Jun 19 10:24:59 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864699, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:59.144441 3848 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:24:59.144886 3848 webserver.cc:460] Webserver started at http://127.19.65.130:37335/ using document root <none> and password file <none>
I20260619 10:24:59.145094 3848 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:59.145159 3848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:59.146373 3848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:59.146880 3863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.147017 3848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:59.147117 3848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "a8fc42d58989417b9fcc29e868c8a2cc"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "554d7fbbc15413db786ba6d7a92aef57"
server_key_iv: "0a93819b94248effcd19f256b037fe3e"
server_key_version: "encryptionkey@0"
I20260619 10:24:59.147451 3848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:59.162896 3848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:59.164762 3848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:59.164909 3848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:59.165148 3848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:59.165428 3848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:59.165480 3848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.165518 3848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:59.165545 3848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.171118 3848 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:39737
I20260619 10:24:59.171151 3976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:39737 every 8 connection(s)
I20260619 10:24:59.171531 3848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:24:59.172499 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.172114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:48317)
0619 10:24:59.172251 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.172346 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20260619 10:24:59.172883 3977 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:43483 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:43483: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
I20260619 10:24:59.173184 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.173099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:47989)
0619 10:24:59.173142 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.173167 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:24:59.173463 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.173395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:58231)
0619 10:24:59.173430 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.173449 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:24:59.173565 3977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:24:59.176987 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3848
I20260619 10:24:59.177253 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:45171
--local_ip_for_outbound_sockets=127.19.65.131
--tserver_master_addrs=127.19.65.190:43483
--webserver_port=41425
--webserver_interface=127.19.65.131
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:24:59.259558 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:24:59.259766 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:24:59.259800 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:24:59.260895 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:24:59.260957 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:24:59.260998 3980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:24:59.262434 3980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:45171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=41425
--tserver_master_addrs=127.19.65.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:24:59.262694 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:24:59.262914 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:24:59.267843 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:59.267946 3980 server_base.cc:1034] running on GCE node
W20260619 10:24:59.267818 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:24:59.267819 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:24:59.268231 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:24:59.268488 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:24:59.269675 3980 hybrid_clock.cc:648] HybridClock initialized: now 1781864699269665 us; error 29 us; skew 500 ppm
Jun 19 10:24:59 dist-test-slave-xd2q krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864699, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:24:59.272121 3980 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:24:59.272575 3980 webserver.cc:460] Webserver started at http://127.19.65.131:41425/ using document root <none> and password file <none>
I20260619 10:24:59.272809 3980 fs_manager.cc:359] Metadata directory not provided
I20260619 10:24:59.272873 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:24:59.274149 3980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:24:59.274721 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.274880 3980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:24:59.274981 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "a1d76052b8be44279a5d0dba594f7b2e"
format_stamp: "Formatted at 2026-06-19 10:24:17 on dist-test-slave-xd2q"
server_key: "d0ba1bee6a19c3e3e15ac366e3f0ffc1"
server_key_iv: "d7fe3b68516173df0885176fd377b3f9"
server_key_version: "encryptionkey@0"
I20260619 10:24:59.275295 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:24:59.287523 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:24:59.289243 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:24:59.289412 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:24:59.289628 3980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:24:59.289914 3980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:24:59.289965 3980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.290005 3980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:24:59.290033 3980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:24:59.295933 3980 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:45171
I20260619 10:24:59.295981 4108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:45171 every 8 connection(s)
I20260619 10:24:59.296289 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:24:59.297283 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.296934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:40973)
0619 10:24:59.297083 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.297150 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20260619 10:24:59.297683 4109 heartbeater.cc:644] Failed to heartbeat to 127.19.65.190:43483 (0 consecutive failures): Network error: Failed to ping master at 127.19.65.190:43483: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
I20260619 10:24:59.297960 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.297876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:36411)
0619 10:24:59.297918 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.297943 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:24:59.298259 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:24:59.298192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:38189)
0619 10:24:59.298226 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 10:24:59.298245 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 10:24:59.298369 4109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:24:59.301872 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 3980
I20260619 10:25:00.111713 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.111466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:54211)
0619 10:25:00.111616 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:00.111651 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260619 10:25:00.159559 4114 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:00.159678 4114 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:25:00.160193 4114 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737), a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
Jun 19 10:25:00 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.130@KRBTEST.COM
Jun 19 10:25:00 dist-test-slave-xd2q krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.131@KRBTEST.COM
I20260619 10:25:00.173488 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.160349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:45171 (local address 127.19.65.129:51475)
0619 10:25:00.160480 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:00.160489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:00.160544 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:00.161287 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:00.161290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:00.161685 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:00.161864 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:00.161870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.163051 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.163052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:00.163881 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:00.163884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.164569 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.164570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:00.164684 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:00.168196 (+ 3512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:00.168201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:00.170177 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:00.171948 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:00.171955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:00.171957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:00.172167 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:00.172497 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:00.172497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:00.172498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:00.172628 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:00.172992 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:00.172998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:00.173123 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 10:25:00.173226 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:25:00.173382 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260619 10:25:00.173501 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.160478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:51475 (local address 127.19.65.131:45171)
0619 10:25:00.160632 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:00.160639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:00.160655 (+ 16us) server_negotiation.cc:407] Connection header received
0619 10:25:00.161054 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:00.161056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:00.161104 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:00.161251 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:00.161985 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.162949 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.163968 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.164472 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.164602 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:00.170288 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:00.170291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:00.170291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:00.170301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:00.171856 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:00.172279 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:00.172283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:00.172283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:00.172392 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:00.172702 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:00.172703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:00.172703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:00.172922 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:00.173009 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:00.173275 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 10:25:00.173391 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":39,"threads_started":1}
I20260619 10:25:00.174018 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.173845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:47717)
0619 10:25:00.173942 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:00.173976 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
W20260619 10:25:00.174358 3730 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:00.174538 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.160322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39737 (local address 127.19.65.129:34401)
0619 10:25:00.160375 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:00.160387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:00.160581 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:00.161287 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:00.161290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:00.161684 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:00.161863 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:00.161869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.163046 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.163047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:00.163883 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:00.163885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.164570 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.164571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:00.164680 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:00.168165 (+ 3485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:00.168174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:00.169790 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:00.173021 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:00.173022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:00.173023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:00.173190 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:00.173436 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:00.173436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:00.173437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:00.173495 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:00.174210 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:00.174211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:00.174273 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 10:25:00.174350 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:25:00.174450 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":79}
I20260619 10:25:00.174700 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.160430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:34401 (local address 127.19.65.130:39737)
0619 10:25:00.160603 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:00.160615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:00.160639 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:25:00.161056 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:00.161058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:00.161104 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:00.161248 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:00.162009 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.162939 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.163989 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:00.164471 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:00.164603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:00.169895 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:00.169899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:00.169900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:00.169928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:00.172936 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:00.173281 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:00.173284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:00.173284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:00.173367 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:00.173812 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:00.173813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:00.173814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:00.174142 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:00.174229 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:00.174406 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 10:25:00.174558 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
W20260619 10:25:00.175336 3732 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:00.175388 3732 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731285bf624a4544bd96b020a65449f7; no voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc
I20260619 10:25:00.175498 4114 raft_consensus.cc:2726] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:25:00.298846 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:00.298659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:49303)
0619 10:25:00.298772 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:00.298812 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260619 10:25:01.112345 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:01.112147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:39253)
0619 10:25:01.112271 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:01.112302 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260619 10:25:01.174554 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:01.174360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:56173)
0619 10:25:01.174476 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:01.174503 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260619 10:25:01.299701 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:01.299474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:54337)
0619 10:25:01.299616 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:01.299654 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260619 10:25:01.756764 4123 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:01.756860 4123 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:25:01.757082 4123 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737), a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
W20260619 10:25:01.757511 3730 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
W20260619 10:25:01.757589 3732 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:01.757674 3732 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731285bf624a4544bd96b020a65449f7; no voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc
I20260619 10:25:01.757761 4123 raft_consensus.cc:2726] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:25:02.112987 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:02.112768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:34215)
0619 10:25:02.112911 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:02.112944 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260619 10:25:02.175355 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:02.175104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:60363)
0619 10:25:02.175224 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:02.175255 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260619 10:25:02.300346 4126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:02.300144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:40183)
0619 10:25:02.300281 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:02.300312 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260619 10:25:03.113735 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:03.113482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:34255)
0619 10:25:03.113654 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:03.113689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260619 10:25:03.175917 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:03.175728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:33943)
0619 10:25:03.175848 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:03.175880 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260619 10:25:03.301007 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:03.300780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:56871)
0619 10:25:03.300922 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:03.300962 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260619 10:25:03.409971 4130 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:03.410056 4130 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:25:03.410269 4130 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737), a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
W20260619 10:25:03.410640 3730 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
W20260619 10:25:03.410704 3732 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:03.410759 3732 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731285bf624a4544bd96b020a65449f7; no voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc
I20260619 10:25:03.410830 4130 raft_consensus.cc:2726] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:25:04.114449 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:04.114243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:58301)
0619 10:25:04.114369 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:04.114405 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260619 10:25:04.176497 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:04.176294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:50673)
0619 10:25:04.176425 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:04.176461 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260619 10:25:04.301623 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:04.301429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:58851)
0619 10:25:04.301554 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:04.301586 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260619 10:25:05.115227 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:05.114982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:53917)
0619 10:25:05.115143 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:05.115183 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260619 10:25:05.177321 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:05.177096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:42047)
0619 10:25:05.177225 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:05.177284 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260619 10:25:05.302470 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:05.302227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:42189)
0619 10:25:05.302385 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:05.302431 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":59,"threads_started":1}
I20260619 10:25:06.115958 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:06.115752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:51429)
0619 10:25:06.115873 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:06.115913 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260619 10:25:06.177990 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:06.177758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:45435)
0619 10:25:06.177928 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:06.177955 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260619 10:25:06.303076 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:06.302890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:55773)
0619 10:25:06.303007 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:06.303041 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260619 10:25:07.116871 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:07.116600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:57721)
0619 10:25:07.116780 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:07.116826 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260619 10:25:07.178681 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:07.178438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:43625)
0619 10:25:07.178594 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:07.178646 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260619 10:25:07.303894 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:07.303650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:54109)
0619 10:25:07.303809 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:07.303852 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260619 10:25:07.891371 4143 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:07.891477 4143 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:25:07.891670 4143 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737), a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
W20260619 10:25:07.892092 3732 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
W20260619 10:25:07.892201 3730 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:07.892294 3730 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731285bf624a4544bd96b020a65449f7; no voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc
I20260619 10:25:07.892391 4143 raft_consensus.cc:2726] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:25:08.117686 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:08.117425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:57535)
0619 10:25:08.117589 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:08.117635 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260619 10:25:08.179390 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:08.179168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:52183)
0619 10:25:08.179311 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:08.179352 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260619 10:25:08.304651 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:08.304442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:37445)
0619 10:25:08.304570 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:08.304612 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260619 10:25:09.118381 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:09.118178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:51703)
0619 10:25:09.118303 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:09.118338 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260619 10:25:09.180045 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:09.179829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:37067)
0619 10:25:09.179955 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:09.179991 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260619 10:25:09.305276 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:09.305087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:37223)
0619 10:25:09.305206 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:09.305239 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260619 10:25:10.119158 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:10.118937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:48905)
0619 10:25:10.119077 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:10.119116 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 10:25:10.180718 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:10.180498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:48963)
0619 10:25:10.180623 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:10.180659 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260619 10:25:10.305861 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:10.305667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:44799)
0619 10:25:10.305793 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:10.305824 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260619 10:25:11.119824 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:11.119623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:34835)
0619 10:25:11.119745 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:11.119779 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260619 10:25:11.181461 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:11.181242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:35837)
0619 10:25:11.181385 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:11.181421 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260619 10:25:11.306478 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:11.306291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:51435)
0619 10:25:11.306411 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:11.306443 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260619 10:25:12.120507 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:12.120213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:51295)
0619 10:25:12.120409 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:12.120446 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":54,"threads_started":1}
I20260619 10:25:12.182057 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:12.181866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:46933)
0619 10:25:12.181984 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:12.182016 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260619 10:25:12.307099 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:12.306890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:55905)
0619 10:25:12.307010 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:12.307042 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260619 10:25:13.121304 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:13.121081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:37585)
0619 10:25:13.121224 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:13.121261 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260619 10:25:13.182811 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:13.182607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:55881)
0619 10:25:13.182729 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:13.182767 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260619 10:25:13.307804 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:13.307580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:49861)
0619 10:25:13.307730 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:13.307763 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260619 10:25:14.122124 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:14.121919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.129:47017)
0619 10:25:14.122048 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:14.122082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260619 10:25:14.135524 4163 raft_consensus.cc:482] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:14.135612 4163 raft_consensus.cc:504] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fc42d58989417b9fcc29e868c8a2cc" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39737 } } peers { permanent_uuid: "a1d76052b8be44279a5d0dba594f7b2e" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 45171 } } peers { permanent_uuid: "731285bf624a4544bd96b020a65449f7" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40083 } }
I20260619 10:25:14.135807 4163 leader_election.cc:288] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737), a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171)
W20260619 10:25:14.136204 3730 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1d76052b8be44279a5d0dba594f7b2e (127.19.65.131:45171): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
W20260619 10:25:14.136308 3732 leader_election.cc:341] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8fc42d58989417b9fcc29e868c8a2cc (127.19.65.130:39737): Not found: Tablet not found: 43f54623d6fc450e9952f45c5bc30d61
I20260619 10:25:14.136346 3732 leader_election.cc:302] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731285bf624a4544bd96b020a65449f7; no voters: a1d76052b8be44279a5d0dba594f7b2e, a8fc42d58989417b9fcc29e868c8a2cc
I20260619 10:25:14.136439 4163 raft_consensus.cc:2726] T 43f54623d6fc450e9952f45c5bc30d61 P 731285bf624a4544bd96b020a65449f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:25:14.183560 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:14.183354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.130:58675)
0619 10:25:14.183483 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:14.183519 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
I20260619 10:25:14.305063 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3679
I20260619 10:25:14.308501 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:14.308290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43483 (local address 127.19.65.131:44513)
0619 10:25:14.308427 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:14.308464 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.190:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260619 10:25:14.309759 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3848
I20260619 10:25:14.314142 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 3980
2026-06-19T10:25:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58310 ms)
[----------] 30 tests from SecurityITest (224780 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:14 dist-test-slave-xd2q krb5kdc[4168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:14 dist-test-slave-xd2q krb5kdc[4168](info): set up 2 sockets
Jun 19 10:25:14 dist-test-slave-xd2q krb5kdc[4168](info): commencing operation
krb5kdc: starting...
W20260619 10:25:15.354915 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:15.466249 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44241
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:15.545013 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:15.545187 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:15.545207 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:15.546373 4181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:15.546411 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:15.546425 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:15.546438 4181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:15.546473 4181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:15.547936 4181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:44241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:15.548205 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:15.548421 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:15.553345 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:15.553503 4181 server_base.cc:1034] running on GCE node
W20260619 10:25:15.553326 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:15.553328 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:15.553802 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:15.554088 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:15.555284 4181 hybrid_clock.cc:648] HybridClock initialized: now 1781864715555277 us; error 36 us; skew 500 ppm
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:15.557749 4181 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:15.558146 4181 webserver.cc:460] Webserver started at http://127.19.65.190:38479/ using document root <none> and password file <none>
I20260619 10:25:15.558357 4181 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:15.558422 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:15.558573 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:15.559608 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "444f4aac8ef14fb7bc2eab212356d01e"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "c5dbb1e00b989931f4255a6ee99c6773"
server_key_iv: "83b1717a1c944f387b4be72ed3388831"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.559984 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "444f4aac8ef14fb7bc2eab212356d01e"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "c5dbb1e00b989931f4255a6ee99c6773"
server_key_iv: "83b1717a1c944f387b4be72ed3388831"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.561261 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:15.561961 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.562134 4181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:15.562232 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "444f4aac8ef14fb7bc2eab212356d01e"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "c5dbb1e00b989931f4255a6ee99c6773"
server_key_iv: "83b1717a1c944f387b4be72ed3388831"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.562309 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:15.572068 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:15.573702 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:15.573855 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:15.577611 4181 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:44241
I20260619 10:25:15.577641 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:44241 every 8 connection(s)
I20260619 10:25:15.577950 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:15.578760 4249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:15.580365 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4181
I20260619 10:25:15.580485 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:15.580682 19718 external_mini_cluster.cc:1428] Setting key eff19bca21b2b31bde0f7044c3b64d59
I20260619 10:25:15.581146 4249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: Bootstrap starting.
I20260619 10:25:15.581719 4249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:15.582068 4249 log.cc:824] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:15.582784 4249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: No bootstrap required, opened a new log
I20260619 10:25:15.584978 4249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } }
I20260619 10:25:15.585106 4249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:15.585129 4249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f4aac8ef14fb7bc2eab212356d01e, State: Initialized, Role: FOLLOWER
I20260619 10:25:15.585258 4249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } }
I20260619 10:25:15.585343 4249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:15.585376 4249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:15.585457 4249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:15.586176 4249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } }
I20260619 10:25:15.586293 4249 leader_election.cc:302] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444f4aac8ef14fb7bc2eab212356d01e; no voters:
I20260619 10:25:15.586470 4249 leader_election.cc:288] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:15.586531 4254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:15.586674 4254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [term 1 LEADER]: Becoming Leader. State: Replica: 444f4aac8ef14fb7bc2eab212356d01e, State: Running, Role: LEADER
I20260619 10:25:15.586736 4249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:15.586802 4254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } }
I20260619 10:25:15.587158 4256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 444f4aac8ef14fb7bc2eab212356d01e. Latest consensus state: current_term: 1 leader_uuid: "444f4aac8ef14fb7bc2eab212356d01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } } }
I20260619 10:25:15.587159 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "444f4aac8ef14fb7bc2eab212356d01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f4aac8ef14fb7bc2eab212356d01e" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 44241 } } }
I20260619 10:25:15.587357 4256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:15.587371 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e [sys.catalog]: This master's current role is: LEADER
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
W20260619 10:25:15.589252 4270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:15.589416 4270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:15.589483 4258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:15.589618 4258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:15.591679 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:15.581183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.19.65.190:44241)
0619 10:25:15.581333 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:15.581340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:15.581363 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:25:15.581785 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:15.581793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:15.581854 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:15.581962 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:15.582699 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.583666 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.584793 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.585215 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.585321 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:15.588189 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:15.588193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:15.588194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:15.588206 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:15.590114 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.590464 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.590468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.590468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.590565 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.590809 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.590810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.590810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.591039 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:15.591148 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:15.591351 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 10:25:15.591507 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260619 10:25:15.595717 4258 catalog_manager.cc:1347] Generated new cluster ID: f2354463ff614887ba3d4216bc073cb6
I20260619 10:25:15.595784 4258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:15.623160 4258 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:15.623710 4258 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:15.625523 4258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: Generated new TSK 0
I20260619 10:25:15.625701 4258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:25:15.657996 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:15.738318 4275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:15.738541 4275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:15.738579 4275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:15.739745 4275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:15.739812 4275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:15.739856 4275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:15.741291 4275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:15.741540 4275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:15.741756 4275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:15.746977 4275 server_base.cc:1034] running on GCE node
W20260619 10:25:15.747016 4283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:15.747023 4280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:15.746955 4281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:15.747398 4275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:15.747664 4275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:15.748852 4275 hybrid_clock.cc:648] HybridClock initialized: now 1781864715748836 us; error 51 us; skew 500 ppm
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:15.751545 4275 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:25:15.752031 4275 webserver.cc:460] Webserver started at http://127.19.65.129:43449/ using document root <none> and password file <none>
I20260619 10:25:15.752246 4275 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:15.752319 4275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:15.752471 4275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:15.753576 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "16616f9ff37942e092de146b43d239e3"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "2492f26af292a001ccc5f3d63c623386"
server_key_iv: "0631b9f70864fc555fa40c69a4139536"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.753953 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "16616f9ff37942e092de146b43d239e3"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "2492f26af292a001ccc5f3d63c623386"
server_key_iv: "0631b9f70864fc555fa40c69a4139536"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.755363 4275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:15.756103 4290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.756274 4275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:15.756373 4275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "16616f9ff37942e092de146b43d239e3"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "2492f26af292a001ccc5f3d63c623386"
server_key_iv: "0631b9f70864fc555fa40c69a4139536"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.756456 4275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:15.764689 4275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:15.766938 4275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:15.767048 4275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:15.767210 4275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:15.767518 4275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:15.767554 4275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.767576 4275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:15.767591 4275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.773655 4275 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:44079
I20260619 10:25:15.773689 4403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:44079 every 8 connection(s)
I20260619 10:25:15.774011 4275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:15.782832 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:15.774726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:56133 (local address 127.19.65.190:44241)
0619 10:25:15.774813 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:15.774815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:15.775060 (+ 245us) server_negotiation.cc:407] Connection header received
0619 10:25:15.775559 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:15.775560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:15.775572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:15.775636 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:15.776493 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.777001 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.777916 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.778119 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.778180 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:15.780205 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:15.780207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:15.780207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:15.780214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:15.781600 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.782065 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.782065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.782066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.782115 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.782313 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.782314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.782314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.782409 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:15.782455 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:15.782676 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 10:25:15.782743 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 10:25:15.782994 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:15.774676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44241 (local address 127.19.65.129:56133)
0619 10:25:15.774914 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:15.774949 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:15.775464 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:15.775686 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:15.775689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:15.776226 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:15.776410 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:15.776414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.777085 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.777086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:15.777830 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:15.777831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.778195 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.778196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:15.778284 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:15.778713 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:15.778721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:15.780101 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:15.781782 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:15.781788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:15.781789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:15.781995 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:15.782171 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:15.782172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:15.782172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:15.782259 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:15.782469 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:15.782473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:15.782573 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 10:25:15.782649 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 10:25:15.782760 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260619 10:25:15.783536 4404 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44241
I20260619 10:25:15.783598 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4275
I20260619 10:25:15.783645 4404 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:15.783701 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:15.783869 4404 heartbeater.cc:511] Master 127.19.65.190:44241 requested a full tablet report, sending...
I20260619 10:25:15.783903 19718 external_mini_cluster.cc:1428] Setting key 0eb8d840d8b88a2be6efd9fc164819ac
I20260619 10:25:15.784453 4212 ts_manager.cc:194] Registered new tserver with Master: 16616f9ff37942e092de146b43d239e3 (127.19.65.129:44079)
I20260619 10:25:15.785066 4212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:56133
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:25:15.808385 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:15.886607 4409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:15.886801 4409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:15.886829 4409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:15.888031 4409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:15.888069 4409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:15.888095 4409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:15.889608 4409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:15.889784 4409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:15.889940 4409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:15.894767 4409 server_base.cc:1034] running on GCE node
W20260619 10:25:15.894683 4415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:15.894685 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:15.894690 4417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:15.895100 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:15.895340 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:15.896494 4409 hybrid_clock.cc:648] HybridClock initialized: now 1781864715896468 us; error 41 us; skew 500 ppm
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:15.898867 4409 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:25:15.899364 4409 webserver.cc:460] Webserver started at http://127.19.65.130:33611/ using document root <none> and password file <none>
I20260619 10:25:15.899600 4409 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:15.899681 4409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:15.899819 4409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:15.900815 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "e029b27fe1114373aff39b7959e9f9ac"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "d5e6617be73e46cf9df4fa7b633dceb9"
server_key_iv: "5acb9a12d4ee8ebfcb8c6913412ae5b2"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.901165 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "e029b27fe1114373aff39b7959e9f9ac"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "d5e6617be73e46cf9df4fa7b633dceb9"
server_key_iv: "5acb9a12d4ee8ebfcb8c6913412ae5b2"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.902406 4409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:15.903134 4424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.903354 4409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:15.903458 4409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "e029b27fe1114373aff39b7959e9f9ac"
format_stamp: "Formatted at 2026-06-19 10:25:15 on dist-test-slave-xd2q"
server_key: "d5e6617be73e46cf9df4fa7b633dceb9"
server_key_iv: "5acb9a12d4ee8ebfcb8c6913412ae5b2"
server_key_version: "encryptionkey@0"
I20260619 10:25:15.903543 4409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:15.915485 4409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:15.917675 4409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:15.917827 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:15.918042 4409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:15.918320 4409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:15.918385 4409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.918421 4409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:15.918447 4409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:15.924207 4409 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:38231
I20260619 10:25:15.924256 4537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:38231 every 8 connection(s)
I20260619 10:25:15.924564 4409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:25:15 dist-test-slave-xd2q krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:15.933048 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:15.925289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:37157 (local address 127.19.65.190:44241)
0619 10:25:15.925346 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:15.925348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:15.925493 (+ 145us) server_negotiation.cc:407] Connection header received
0619 10:25:15.925938 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:15.925939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:15.925961 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:15.926032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:15.926833 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.927330 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.928210 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.928381 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.928429 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:15.930509 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:15.930511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:15.930511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:15.930517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:15.931723 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.932079 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.932080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.932080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.932127 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:15.932441 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:15.932442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:15.932442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:15.932543 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:15.932592 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:15.932844 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 10:25:15.932917 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:25:15.933059 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:15.925246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44241 (local address 127.19.65.130:37157)
0619 10:25:15.925394 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:15.925431 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:15.925855 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:15.926076 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:15.926078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:15.926580 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:15.926758 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:15.926762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.927399 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.927400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:15.928140 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:15.928142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:15.928449 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:15.928450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:15.928548 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:15.929090 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:15.929097 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:15.930427 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:15.931792 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:15.931799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:15.931801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:15.932017 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:15.932252 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:15.932254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:15.932254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:15.932380 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:15.932600 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:15.932604 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:15.932706 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 10:25:15.932789 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:25:15.932911 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260619 10:25:15.933131 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4409
I20260619 10:25:15.933218 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:15.933418 19718 external_mini_cluster.cc:1428] Setting key ffcc4b51cd146ce5b7ded0514917e493
I20260619 10:25:15.933867 4538 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44241
I20260619 10:25:15.933987 4538 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:15.934151 4538 heartbeater.cc:511] Master 127.19.65.190:44241 requested a full tablet report, sending...
I20260619 10:25:15.935863 4212 ts_manager.cc:194] Registered new tserver with Master: e029b27fe1114373aff39b7959e9f9ac (127.19.65.130:38231)
I20260619 10:25:15.936385 4212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:37157
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:25:15.958360 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:16.037540 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:16.037751 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:16.037789 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:16.038887 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:16.038955 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:16.038997 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:16.040479 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:16.040733 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:16.040946 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:16.045826 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.045838 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.045878 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.045912 4543 server_base.cc:1034] running on GCE node
I20260619 10:25:16.046449 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:16.046710 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:16.047885 4543 hybrid_clock.cc:648] HybridClock initialized: now 1781864716047880 us; error 34 us; skew 500 ppm
Jun 19 10:25:16 dist-test-slave-xd2q krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864716, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:16.050300 4543 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:25:16.050747 4543 webserver.cc:460] Webserver started at http://127.19.65.131:34569/ using document root <none> and password file <none>
I20260619 10:25:16.050951 4543 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:16.051010 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:16.051173 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:16.052196 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "7c0c0edf7da241c6a2f87b94ff7988f5"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "67b9b3a814b8f223f9d9cdea7335188b"
server_key_iv: "6c69a212c2d23fea4de00e3ae9d7d082"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.052569 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "7c0c0edf7da241c6a2f87b94ff7988f5"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "67b9b3a814b8f223f9d9cdea7335188b"
server_key_iv: "6c69a212c2d23fea4de00e3ae9d7d082"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.053803 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:16.054477 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.054636 4543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:16.054729 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "7c0c0edf7da241c6a2f87b94ff7988f5"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "67b9b3a814b8f223f9d9cdea7335188b"
server_key_iv: "6c69a212c2d23fea4de00e3ae9d7d082"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.054801 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:16.075173 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:16.077834 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:16.077979 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:16.078209 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:16.078501 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:16.078532 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.078581 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:16.078606 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.084609 4543 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:38875
I20260619 10:25:16.084699 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:38875 every 8 connection(s)
I20260619 10:25:16.084992 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:25:16 dist-test-slave-xd2q krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864716, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:16.092980 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4543
I20260619 10:25:16.093082 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:16.093278 19718 external_mini_cluster.cc:1428] Setting key 4d9399823e92d809d3f3e7c0591f32a1
I20260619 10:25:16.094803 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.085705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38197 (local address 127.19.65.190:44241)
0619 10:25:16.085747 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.085749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.085980 (+ 231us) server_negotiation.cc:407] Connection header received
0619 10:25:16.086518 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.086519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.086532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.086582 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.087612 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.088200 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.089310 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.089510 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.089614 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.091734 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.091736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.091736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:16.091744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.093293 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:16.093788 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:16.093789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:16.093790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:16.093850 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:16.094125 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:16.094126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:16.094126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:16.094256 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.094353 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.094621 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.094706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:16.094974 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.085692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:44241 (local address 127.19.65.131:38197)
0619 10:25:16.085869 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.085902 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.086379 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.086658 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.086661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.087280 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:16.087520 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.087525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.088305 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.088307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.089223 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.089226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.089621 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.089622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.089740 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.090166 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:16.090174 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:16.091636 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:16.093414 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:16.093421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:16.093424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:16.093717 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:16.093931 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:16.093932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:16.093933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:16.094049 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:16.094337 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:16.094343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:16.094481 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.094586 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.094746 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260619 10:25:16.095444 4672 heartbeater.cc:347] Connected to a master server at 127.19.65.190:44241
I20260619 10:25:16.095517 4672 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:16.095695 4672 heartbeater.cc:511] Master 127.19.65.190:44241 requested a full tablet report, sending...
I20260619 10:25:16.096033 4212 ts_manager.cc:194] Registered new tserver with Master: 7c0c0edf7da241c6a2f87b94ff7988f5 (127.19.65.131:38875)
I20260619 10:25:16.096344 4212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:38197
I20260619 10:25:16.104931 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:16.112119 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.106420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.19.65.190:44241)
0619 10:25:16.106491 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.106493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.106581 (+ 88us) server_negotiation.cc:407] Connection header received
0619 10:25:16.106606 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.106607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.106616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.106671 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.107240 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.107794 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.108482 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.108665 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.108730 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.109778 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.109779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.109779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:16.109785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.110976 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:16.111280 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:16.111281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:16.111281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:16.111379 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:16.111622 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:16.111623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:16.111623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:16.111771 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.111869 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.111956 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.112015 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:25:16.113618 4212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:16.115216 4212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:16.121431 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:38231 (local address 127.0.0.1:47306)
0619 10:25:16.117589 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.117630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.117700 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.118066 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.118067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.118069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.118114 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.118117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.119488 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.119489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.120436 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.120438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121151 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.121229 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121241 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.121308 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.121360 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":34,"thread_start_us":48,"threads_started":1}
I20260619 10:25:16.121443 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.19.65.130:38231)
0619 10:25:16.117862 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.117866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.117875 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:16.117909 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.117910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.117924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.117990 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.118191 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.119357 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.120535 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121063 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121150 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121206 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.121331 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.121369 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":86,"threads_started":1}
I20260619 10:25:16.121615 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.19.65.131:38875)
0619 10:25:16.117825 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.117829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.117838 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:16.117887 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.117889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.117900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.117985 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.118147 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.118888 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.120375 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121131 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121199 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121248 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.121474 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.121519 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260619 10:25:16.121773 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.19.65.129:44079)
0619 10:25:16.117562 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.117566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.117829 (+ 263us) server_negotiation.cc:407] Connection header received
0619 10:25:16.117893 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.117896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.117911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.118192 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.118275 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.119382 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.120671 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121342 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121402 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121448 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.121610 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.121654 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260619 10:25:16.121694 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44079 (local address 127.0.0.1:40184)
0619 10:25:16.117650 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.117662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.117714 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.118099 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.118100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.118101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.118129 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.118130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.119495 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.119496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.120581 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.120596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121480 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.121529 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121533 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.121588 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.121622 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":29,"thread_start_us":58,"threads_started":1}
I20260619 10:25:16.122076 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.117477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:38875 (local address 127.0.0.1:35652)
0619 10:25:16.117651 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.117662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.117714 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.118003 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.118006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.118009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.118067 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.118068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.119031 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.119032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.120290 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.120293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.121282 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.121283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.121340 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.121345 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.121394 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.121431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1,"thread_start_us":43,"threads_started":1}
I20260619 10:25:16.122510 4472 tablet_service.cc:1467] Processing CreateTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 (DEFAULT_TABLE table=test-table [id=6d54b631d31d47b899e693d6be597d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.122607 4338 tablet_service.cc:1467] Processing CreateTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 (DEFAULT_TABLE table=test-table [id=6d54b631d31d47b899e693d6be597d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.122865 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c43d4177a40cbba27d6e0f4b71d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.123018 4338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c43d4177a40cbba27d6e0f4b71d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.125264 4686 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Bootstrap starting.
I20260619 10:25:16.125877 4687 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: Bootstrap starting.
I20260619 10:25:16.126061 4606 tablet_service.cc:1467] Processing CreateTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 (DEFAULT_TABLE table=test-table [id=6d54b631d31d47b899e693d6be597d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.126150 4686 tablet_bootstrap.cc:654] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.126355 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c43d4177a40cbba27d6e0f4b71d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.126485 4687 tablet_bootstrap.cc:654] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.126746 4686 log.cc:824] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.127063 4687 log.cc:824] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.127612 4686 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: No bootstrap required, opened a new log
I20260619 10:25:16.127682 4686 ts_tablet_manager.cc:1397] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:16.127825 4687 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: No bootstrap required, opened a new log
I20260619 10:25:16.127884 4687 ts_tablet_manager.cc:1397] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:16.129281 4690 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Bootstrap starting.
I20260619 10:25:16.129650 4686 raft_consensus.cc:348] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.129801 4690 tablet_bootstrap.cc:654] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.129820 4686 raft_consensus.cc:374] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.129858 4686 raft_consensus.cc:729] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e029b27fe1114373aff39b7959e9f9ac, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.129901 4687 raft_consensus.cc:348] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.130004 4686 consensus_queue.cc:260] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.130059 4687 raft_consensus.cc:374] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.130095 4687 raft_consensus.cc:729] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16616f9ff37942e092de146b43d239e3, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.130254 4686 ts_tablet_manager.cc:1428] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:16.130227 4687 consensus_queue.cc:260] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.130379 4538 heartbeater.cc:503] Master 127.19.65.190:44241 was elected leader, sending a full tablet report...
I20260619 10:25:16.130395 4690 log.cc:824] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.130467 4687 ts_tablet_manager.cc:1428] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:16.130568 4404 heartbeater.cc:503] Master 127.19.65.190:44241 was elected leader, sending a full tablet report...
I20260619 10:25:16.131402 4690 tablet_bootstrap.cc:492] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: No bootstrap required, opened a new log
I20260619 10:25:16.131466 4690 ts_tablet_manager.cc:1397] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:16.132906 4690 raft_consensus.cc:348] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.133108 4690 raft_consensus.cc:374] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.133157 4690 raft_consensus.cc:729] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0c0edf7da241c6a2f87b94ff7988f5, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.133263 4690 consensus_queue.cc:260] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.133548 4690 ts_tablet_manager.cc:1428] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:16.133701 4672 heartbeater.cc:503] Master 127.19.65.190:44241 was elected leader, sending a full tablet report...
W20260619 10:25:16.175092 4539 tablet.cc:2369] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:16.197458 4691 raft_consensus.cc:482] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:16.197638 4691 raft_consensus.cc:504] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.198053 4691 leader_election.cc:288] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0c0edf7da241c6a2f87b94ff7988f5 (127.19.65.131:38875), 16616f9ff37942e092de146b43d239e3 (127.19.65.129:44079)
I20260619 10:25:16.200958 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.198324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:42553 (local address 127.19.65.131:38875)
0619 10:25:16.198370 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.198372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.198379 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:16.198403 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.198404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.198414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.198461 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.198667 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.199195 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200165 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.200582 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200637 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.200644 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.200828 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.200875 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:25:16.200980 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.198229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:38875 (local address 127.19.65.130:42553)
0619 10:25:16.198271 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.198285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.198333 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.198484 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.198485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.198487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.198562 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.198564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.199278 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.199279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.200071 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.200074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.200664 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.200749 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.200753 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.200811 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.200895 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260619 10:25:16.201022 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.198412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:33823 (local address 127.19.65.129:44079)
0619 10:25:16.198479 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.198481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.198509 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:25:16.198583 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.198584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.198595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.198650 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.198804 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.199262 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200265 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.200662 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200734 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.200741 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.200910 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.200951 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:25:16.201272 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.198254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44079 (local address 127.19.65.130:33823)
0619 10:25:16.198460 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.198469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.198514 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.198667 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.198668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.198669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.198734 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.198736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.199406 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.199407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.200175 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.200177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.200745 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.200746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.200825 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.200829 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.200884 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.200920 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":9,"thread_start_us":112,"threads_started":1}
I20260619 10:25:16.201385 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c43d4177a40cbba27d6e0f4b71d49" candidate_uuid: "e029b27fe1114373aff39b7959e9f9ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" is_pre_election: true
I20260619 10:25:16.201550 4626 raft_consensus.cc:2454] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e029b27fe1114373aff39b7959e9f9ac in term 0.
I20260619 10:25:16.201628 4358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c43d4177a40cbba27d6e0f4b71d49" candidate_uuid: "e029b27fe1114373aff39b7959e9f9ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16616f9ff37942e092de146b43d239e3" is_pre_election: true
I20260619 10:25:16.201748 4427 leader_election.cc:302] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0c0edf7da241c6a2f87b94ff7988f5, e029b27fe1114373aff39b7959e9f9ac; no voters:
I20260619 10:25:16.201778 4358 raft_consensus.cc:2454] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e029b27fe1114373aff39b7959e9f9ac in term 0.
I20260619 10:25:16.201869 4691 raft_consensus.cc:2781] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:16.201923 4691 raft_consensus.cc:482] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:16.201946 4691 raft_consensus.cc:3037] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.202641 4691 raft_consensus.cc:504] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.202773 4691 leader_election.cc:288] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [CANDIDATE]: Term 1 election: Requested vote from peers 7c0c0edf7da241c6a2f87b94ff7988f5 (127.19.65.131:38875), 16616f9ff37942e092de146b43d239e3 (127.19.65.129:44079)
I20260619 10:25:16.202943 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c43d4177a40cbba27d6e0f4b71d49" candidate_uuid: "e029b27fe1114373aff39b7959e9f9ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5"
I20260619 10:25:16.203022 4626 raft_consensus.cc:3037] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.202992 4358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c43d4177a40cbba27d6e0f4b71d49" candidate_uuid: "e029b27fe1114373aff39b7959e9f9ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16616f9ff37942e092de146b43d239e3"
I20260619 10:25:16.203097 4358 raft_consensus.cc:3037] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.203804 4626 raft_consensus.cc:2454] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e029b27fe1114373aff39b7959e9f9ac in term 1.
I20260619 10:25:16.203887 4358 raft_consensus.cc:2454] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e029b27fe1114373aff39b7959e9f9ac in term 1.
I20260619 10:25:16.203962 4427 leader_election.cc:302] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0c0edf7da241c6a2f87b94ff7988f5, e029b27fe1114373aff39b7959e9f9ac; no voters:
I20260619 10:25:16.204048 4691 raft_consensus.cc:2781] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:16.204205 4691 raft_consensus.cc:686] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 1 LEADER]: Becoming Leader. State: Replica: e029b27fe1114373aff39b7959e9f9ac, State: Running, Role: LEADER
I20260619 10:25:16.204282 4691 consensus_queue.cc:237] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } }
I20260619 10:25:16.205097 4213 catalog_manager.cc:5495] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e029b27fe1114373aff39b7959e9f9ac (127.19.65.130). New cstate: current_term: 1 leader_uuid: "e029b27fe1114373aff39b7959e9f9ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e029b27fe1114373aff39b7959e9f9ac" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 } health_report { overall_health: UNKNOWN } } }
I20260619 10:25:16.215683 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.213563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.19.65.130:38231)
0619 10:25:16.213620 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.213622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.213631 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:16.213678 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.213679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.213696 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.213769 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:16.213876 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.214313 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.215017 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.215198 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.215253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.215426 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:16.215516 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:16.215581 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.215602 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.215619 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:16.217415 4626 raft_consensus.cc:1261] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 1 FOLLOWER]: Refusing update from remote peer e029b27fe1114373aff39b7959e9f9ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:16.217415 4358 raft_consensus.cc:1261] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 1 FOLLOWER]: Refusing update from remote peer e029b27fe1114373aff39b7959e9f9ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:16.217792 4696 consensus_queue.cc:1035] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0c0edf7da241c6a2f87b94ff7988f5" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:16.217900 4691 consensus_queue.cc:1035] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "16616f9ff37942e092de146b43d239e3" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:16.220062 4699 mvcc.cc:205] Tried to move back new op lower bound from 7298517877623373824 to 7298517877573226496. Current Snapshot: MvccSnapshot[applied={T|T < 7298517877623373824}]
I20260619 10:25:16.221104 4700 mvcc.cc:205] Tried to move back new op lower bound from 7298517877623373824 to 7298517877573226496. Current Snapshot: MvccSnapshot[applied={T|T < 7298517877623373824}]
I20260619 10:25:16.222330 4212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:16.222417 4212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:16.223132 4212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 444f4aac8ef14fb7bc2eab212356d01e: Sending DeleteTablet for 3 replicas of tablet 558c43d4177a40cbba27d6e0f4b71d49
I20260619 10:25:16.223438 4472 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:47306
I20260619 10:25:16.223456 4338 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:40184
I20260619 10:25:16.223487 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c43d4177a40cbba27d6e0f4b71d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:35652
I20260619 10:25:16.223680 4709 tablet_replica.cc:330] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3: stopping tablet replica
I20260619 10:25:16.223762 4709 raft_consensus.cc:2229] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:16.223861 4709 raft_consensus.cc:2258] T 558c43d4177a40cbba27d6e0f4b71d49 P 16616f9ff37942e092de146b43d239e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:16.223877 4708 tablet_replica.cc:330] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: stopping tablet replica
I20260619 10:25:16.223932 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4275
I20260619 10:25:16.224048 4708 raft_consensus.cc:2229] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:16.224176 4708 raft_consensus.cc:2258] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:16.224475 4710 tablet_replica.cc:330] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: stopping tablet replica
I20260619 10:25:16.224570 4710 raft_consensus.cc:2229] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:16.224673 4710 raft_consensus.cc:2258] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:16.224810 4708 ts_tablet_manager.cc:1905] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:16.226408 4708 ts_tablet_manager.cc:1918] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:16.226496 4708 log.cc:1196] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/558c43d4177a40cbba27d6e0f4b71d49
I20260619 10:25:16.226462 4710 ts_tablet_manager.cc:1905] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:16.226795 4708 ts_tablet_manager.cc:1939] T 558c43d4177a40cbba27d6e0f4b71d49 P e029b27fe1114373aff39b7959e9f9ac: Deleting consensus metadata
I20260619 10:25:16.227975 4199 catalog_manager.cc:4841] TS e029b27fe1114373aff39b7959e9f9ac (127.19.65.130:38231): tablet 558c43d4177a40cbba27d6e0f4b71d49 (table test-table [id=6d54b631d31d47b899e693d6be597d6a]) successfully deleted
I20260619 10:25:16.228675 4710 ts_tablet_manager.cc:1918] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:16.228745 4710 log.cc:1196] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/558c43d4177a40cbba27d6e0f4b71d49
I20260619 10:25:16.228998 4710 ts_tablet_manager.cc:1939] T 558c43d4177a40cbba27d6e0f4b71d49 P 7c0c0edf7da241c6a2f87b94ff7988f5: Deleting consensus metadata
I20260619 10:25:16.229362 4199 catalog_manager.cc:4841] TS 7c0c0edf7da241c6a2f87b94ff7988f5 (127.19.65.131:38875): tablet 558c43d4177a40cbba27d6e0f4b71d49 (table test-table [id=6d54b631d31d47b899e693d6be597d6a]) successfully deleted
W20260619 10:25:16.231338 4198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:44079 (error 108)
I20260619 10:25:16.231527 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4409
I20260619 10:25:16.232205 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.231662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:44079 (local address 127.0.0.1:40194)
0619 10:25:16.232142 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.232179 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:44079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462}
W20260619 10:25:16.232326 4198 catalog_manager.cc:4584] TS 16616f9ff37942e092de146b43d239e3 (127.19.65.129:44079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 558c43d4177a40cbba27d6e0f4b71d49: Network error: Client connection negotiation failed: client connection to 127.19.65.129:44079: connect: Connection refused (error 111)
I20260619 10:25:16.236876 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4543
I20260619 10:25:16.241642 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4181
2026-06-19T10:25:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T10:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:16Z Disabled control of system clock
I20260619 10:25:16.268432 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42231
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:16.348284 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:16.348491 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:16.348513 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:16.349539 4715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:16.349575 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:16.349591 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:16.349603 4715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:16.349617 4715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:16.350828 4715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:42231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:16.351007 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:16.351222 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:16.355734 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.355904 4715 server_base.cc:1034] running on GCE node
W20260619 10:25:16.355795 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.355747 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.356221 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:16.356503 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:16.357672 4715 hybrid_clock.cc:648] HybridClock initialized: now 1781864716357650 us; error 33 us; skew 500 ppm
I20260619 10:25:16.358916 4715 webserver.cc:460] Webserver started at http://127.19.65.190:32897/ using document root <none> and password file <none>
I20260619 10:25:16.359148 4715 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:16.359218 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:16.359396 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:16.360386 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "4422c998e2f28d3541d0027fb1ccc914"
server_key_iv: "f330798c8fe4b228063b46ec07e00eef"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.360793 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "4422c998e2f28d3541d0027fb1ccc914"
server_key_iv: "f330798c8fe4b228063b46ec07e00eef"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.362124 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:16.362895 4729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.363178 4715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:16.363273 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "4422c998e2f28d3541d0027fb1ccc914"
server_key_iv: "f330798c8fe4b228063b46ec07e00eef"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.363396 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:16.379979 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:16.380246 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:16.380390 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:16.384114 4715 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:42231
I20260619 10:25:16.384188 4781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:42231 every 8 connection(s)
I20260619 10:25:16.384517 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:16.385505 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.387673 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: Bootstrap starting.
I20260619 10:25:16.388166 4782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.388402 4782 log.cc:824] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.389014 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: No bootstrap required, opened a new log
I20260619 10:25:16.390516 4782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } }
I20260619 10:25:16.390645 4782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.390668 4782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b39f3bb25b549fbb7709fb5c6b17d8a, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.390815 4782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } }
I20260619 10:25:16.390895 4782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:16.390918 4782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:16.390955 4782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.391467 4782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } }
I20260619 10:25:16.391553 4782 leader_election.cc:302] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b39f3bb25b549fbb7709fb5c6b17d8a; no voters:
I20260619 10:25:16.391672 4782 leader_election.cc:288] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:16.391727 4785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:16.391898 4785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [term 1 LEADER]: Becoming Leader. State: Replica: 3b39f3bb25b549fbb7709fb5c6b17d8a, State: Running, Role: LEADER
I20260619 10:25:16.391913 4782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:16.392001 4785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } }
I20260619 10:25:16.392185 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4715
I20260619 10:25:16.392304 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:16.392586 19718 external_mini_cluster.cc:1428] Setting key 6e08e3b2c8d8a71f6bfa28559be6e33e
I20260619 10:25:16.393175 4787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b39f3bb25b549fbb7709fb5c6b17d8a. Latest consensus state: current_term: 1 leader_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } } }
I20260619 10:25:16.393294 4787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:16.393491 4802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:16.393501 4786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b39f3bb25b549fbb7709fb5c6b17d8a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 42231 } } }
I20260619 10:25:16.393759 4786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:16.394164 4802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:16.395705 4802 catalog_manager.cc:1347] Generated new cluster ID: b3716c855d5249d8ba87c1059e258bce
I20260619 10:25:16.395756 4802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:16.398564 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.393044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.19.65.190:42231)
0619 10:25:16.393459 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.393467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.393592 (+ 125us) server_negotiation.cc:407] Connection header received
0619 10:25:16.393710 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.393717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.393783 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.393925 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.394197 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.395402 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.396480 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.397078 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.397188 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.397894 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.397898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.397898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:16.398090 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.398129 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:16.398143 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.398204 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.398308 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.398376 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":75,"threads_started":1}
I20260619 10:25:16.419200 4802 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:16.419922 4802 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:16.423583 4802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: Generated new TSK 0
I20260619 10:25:16.423743 4802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:16.426689 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--builtin_ntp_servers=127.19.65.148:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:16.507150 4806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:16.507401 4806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:16.507457 4806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:16.508558 4806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:16.508618 4806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:16.508657 4806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:16.510000 4806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:16.510259 4806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:16.510563 4806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:16.515496 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.515471 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.515515 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.516305 4806 server_base.cc:1034] running on GCE node
I20260619 10:25:16.516532 4806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:16.516790 4806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:16.517939 4806 hybrid_clock.cc:648] HybridClock initialized: now 1781864716517924 us; error 31 us; skew 500 ppm
I20260619 10:25:16.519170 4806 webserver.cc:460] Webserver started at http://127.19.65.129:44015/ using document root <none> and password file <none>
I20260619 10:25:16.519469 4806 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:16.519543 4806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:16.519695 4806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:16.520787 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "944d77b7927a47b8bb8171c61bb14e51"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "0953bc243554eb680620926fa75b7213"
server_key_iv: "b1694702aa7e76f77968480bca04476e"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.521148 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "944d77b7927a47b8bb8171c61bb14e51"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "0953bc243554eb680620926fa75b7213"
server_key_iv: "b1694702aa7e76f77968480bca04476e"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.522418 4806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:16.523188 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.523383 4806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:16.523464 4806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "944d77b7927a47b8bb8171c61bb14e51"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "0953bc243554eb680620926fa75b7213"
server_key_iv: "b1694702aa7e76f77968480bca04476e"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.523553 4806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:16.542291 4806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:16.542582 4806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:16.542722 4806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:16.542964 4806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:16.543284 4806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:16.543380 4806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.543413 4806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:16.543443 4806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.549492 4806 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:38253
I20260619 10:25:16.549604 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:38253 every 8 connection(s)
I20260619 10:25:16.549904 4806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:16.551491 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4806
I20260619 10:25:16.551609 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:16.551812 19718 external_mini_cluster.cc:1428] Setting key 2379960e1f7ec1422c0ab8458d715839
I20260619 10:25:16.552898 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--builtin_ntp_servers=127.19.65.148:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:25:16.556393 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.550670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:59221 (local address 127.19.65.190:42231)
0619 10:25:16.550875 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.550878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.551065 (+ 187us) server_negotiation.cc:407] Connection header received
0619 10:25:16.551731 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.551732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.551753 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.551825 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.552737 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.553527 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.554974 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.555161 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.555219 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.556136 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.556137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.556137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:16.556141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.556157 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:16.556160 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.556199 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.556315 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.556338 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260619 10:25:16.556708 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.550622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42231 (local address 127.19.65.129:59221)
0619 10:25:16.550926 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.550976 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.551576 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.551912 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.551917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.551927 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:16.552646 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.552653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.553664 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.553665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.554868 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.554873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.555237 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.555238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.555418 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.555968 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:16.555969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:16.556007 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:16.556037 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:16.556224 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:16.556229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:16.556232 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.556321 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.556436 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":124,"threads_started":1}
I20260619 10:25:16.557314 4934 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42231
I20260619 10:25:16.557422 4934 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:16.557638 4934 heartbeater.cc:511] Master 127.19.65.190:42231 requested a full tablet report, sending...
I20260619 10:25:16.558135 4746 ts_manager.cc:194] Registered new tserver with Master: 944d77b7927a47b8bb8171c61bb14e51 (127.19.65.129:38253)
I20260619 10:25:16.558738 4746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.129:59221
W20260619 10:25:16.634593 4937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:16.634784 4937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:16.634807 4937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:16.635986 4937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:16.636029 4937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:16.636086 4937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:16.637414 4937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:16.637666 4937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:16.637993 4937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:16.642803 4945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.642778 4942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.642779 4943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.642982 4937 server_base.cc:1034] running on GCE node
I20260619 10:25:16.643338 4937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:16.643579 4937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:16.644749 4937 hybrid_clock.cc:648] HybridClock initialized: now 1781864716644738 us; error 29 us; skew 500 ppm
I20260619 10:25:16.646011 4937 webserver.cc:460] Webserver started at http://127.19.65.130:43899/ using document root <none> and password file <none>
I20260619 10:25:16.646219 4937 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:16.646274 4937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:16.646374 4937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:16.647456 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "a37270f5aaaf4984aa1c158e51a349f2"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "a870a9a86ec1e7f894127b876fa92d5d"
server_key_iv: "cab13f890638449b48a99e4b3289bd6a"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.647778 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "a37270f5aaaf4984aa1c158e51a349f2"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "a870a9a86ec1e7f894127b876fa92d5d"
server_key_iv: "cab13f890638449b48a99e4b3289bd6a"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.649016 4937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:16.649667 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.649847 4937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:16.649941 4937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "a37270f5aaaf4984aa1c158e51a349f2"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "a870a9a86ec1e7f894127b876fa92d5d"
server_key_iv: "cab13f890638449b48a99e4b3289bd6a"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.650050 4937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:16.663980 4937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:16.664292 4937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:16.664445 4937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:16.664698 4937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:16.665069 4937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:16.665115 4937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.665150 4937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:16.665197 4937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.671448 4937 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:41173
I20260619 10:25:16.671526 5064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:41173 every 8 connection(s)
I20260619 10:25:16.671835 4937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:16.676960 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.672546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:47237 (local address 127.19.65.190:42231)
0619 10:25:16.672607 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.672609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.672820 (+ 211us) server_negotiation.cc:407] Connection header received
0619 10:25:16.673344 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.673346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.673359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.673406 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.674086 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.674568 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.675544 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.675738 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.675812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.676644 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.676646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.676646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:16.676651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.676668 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:16.676672 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.676752 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.676846 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.676877 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:25:16.677145 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.672534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42231 (local address 127.19.65.130:47237)
0619 10:25:16.672704 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.672734 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.673235 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.673490 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.673499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.673518 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:16.673981 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.673985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.674663 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.674664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.675420 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.675421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.675817 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.675818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.675951 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.676468 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:16.676468 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:16.676506 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:16.676525 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:16.676753 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:16.676759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:16.676762 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.676846 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.676939 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260619 10:25:16.677623 5065 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42231
I20260619 10:25:16.677693 5065 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:16.677874 5065 heartbeater.cc:511] Master 127.19.65.190:42231 requested a full tablet report, sending...
I20260619 10:25:16.678277 4746 ts_manager.cc:194] Registered new tserver with Master: a37270f5aaaf4984aa1c158e51a349f2 (127.19.65.130:41173)
I20260619 10:25:16.678421 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 4937
I20260619 10:25:16.678524 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:16.678718 19718 external_mini_cluster.cc:1428] Setting key 825a838244ebcdd2be3851ad45830777
I20260619 10:25:16.678772 4746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.130:47237
I20260619 10:25:16.679643 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--builtin_ntp_servers=127.19.65.148:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:16.757149 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:16.757313 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:16.757332 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:16.758381 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:16.758419 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:16.758476 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:16.759836 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:16.760051 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:16.760309 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:16.765074 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.765085 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:16.765120 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:16.765396 5068 server_base.cc:1034] running on GCE node
I20260619 10:25:16.765816 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:16.766083 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:16.767226 5068 hybrid_clock.cc:648] HybridClock initialized: now 1781864716767199 us; error 40 us; skew 500 ppm
I20260619 10:25:16.768705 5068 webserver.cc:460] Webserver started at http://127.19.65.131:43555/ using document root <none> and password file <none>
I20260619 10:25:16.768926 5068 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:16.768994 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:16.769120 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:16.770179 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "71d0c7e826854afd9816bcff953a5ef0"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "9b1db952728661b4aedba2ea0428a5a5"
server_key_iv: "fd526fc7772046cd3e58778146a96040"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.770524 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "71d0c7e826854afd9816bcff953a5ef0"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "9b1db952728661b4aedba2ea0428a5a5"
server_key_iv: "fd526fc7772046cd3e58778146a96040"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.772039 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:16.772737 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.772925 5068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:16.773021 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "71d0c7e826854afd9816bcff953a5ef0"
format_stamp: "Formatted at 2026-06-19 10:25:16 on dist-test-slave-xd2q"
server_key: "9b1db952728661b4aedba2ea0428a5a5"
server_key_iv: "fd526fc7772046cd3e58778146a96040"
server_key_version: "encryptionkey@0"
I20260619 10:25:16.773105 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:16.785247 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:16.785504 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:16.785640 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:16.785838 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:16.786155 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:16.786211 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.786249 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:16.786275 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:16.792199 5068 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41409
I20260619 10:25:16.792244 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:41409 every 8 connection(s)
I20260619 10:25:16.792568 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:16.793843 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5068
I20260619 10:25:16.793962 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:16.794163 19718 external_mini_cluster.cc:1428] Setting key b137937858ac4b9e84f188c02e028f8f
I20260619 10:25:16.797966 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.793244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:38663 (local address 127.19.65.190:42231)
0619 10:25:16.793290 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.793293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.793576 (+ 283us) server_negotiation.cc:407] Connection header received
0619 10:25:16.794388 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.794390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.794405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.794472 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.795337 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.795820 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.796734 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.796910 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.796961 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.797760 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.797761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.797761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:16.797764 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.797774 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:16.797778 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.797817 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.797889 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.797910 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:16.798154 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.793261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:42231 (local address 127.19.65.131:38663)
0619 10:25:16.793442 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.793480 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.794213 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.794552 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.794555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.794566 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:16.795208 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.795213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.795905 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.795905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.796659 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.796661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.796983 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.796984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.797080 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.797640 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:16.797640 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:16.797667 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:16.797688 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:16.797825 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:16.797827 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:16.797830 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.797876 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.797947 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260619 10:25:16.798538 5196 heartbeater.cc:347] Connected to a master server at 127.19.65.190:42231
I20260619 10:25:16.798637 5196 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:16.798763 5196 heartbeater.cc:511] Master 127.19.65.190:42231 requested a full tablet report, sending...
I20260619 10:25:16.799108 4746 ts_manager.cc:194] Registered new tserver with Master: 71d0c7e826854afd9816bcff953a5ef0 (127.19.65.131:41409)
I20260619 10:25:16.799515 4746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.131:38663
I20260619 10:25:16.805757 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:16.809581 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.807137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.19.65.190:42231)
0619 10:25:16.807191 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.807195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.807332 (+ 137us) server_negotiation.cc:407] Connection header received
0619 10:25:16.807376 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.807377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.807386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.807430 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:16.807536 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.807942 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.808632 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.808790 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.808835 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.809412 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:16.809413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:16.809413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:16.809416 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:16.809428 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:16.809431 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:16.809471 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.809516 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.809535 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:25:16.811038 4746 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:16.812595 4746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:16.818975 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.815132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.19.65.131:41409)
0619 10:25:16.815278 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.815286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.815333 (+ 47us) server_negotiation.cc:407] Connection header received
0619 10:25:16.815369 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.815371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.815384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.815479 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.815583 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.816668 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.817909 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.818600 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.818705 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.818789 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.818890 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.818919 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260619 10:25:16.818995 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.814832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41409 (local address 127.0.0.1:57586)
0619 10:25:16.814953 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.814962 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.815008 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.815480 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.815481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.815484 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.815519 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.815521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.816740 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.816741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.817828 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.817829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.818707 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.818708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.818778 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.818790 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.818880 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.818931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260619 10:25:16.819375 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.815007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38253 (local address 127.0.0.1:52474)
0619 10:25:16.815109 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.815118 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.815158 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.815679 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.815680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.815682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.815717 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.815718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.816572 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.816573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.817826 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.817828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.819140 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.819141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.819204 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.819208 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.819259 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.819310 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260619 10:25:16.819428 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.815126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.19.65.129:38253)
0619 10:25:16.815463 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.815468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.815476 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:16.815513 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.815515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.815529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.815635 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.815826 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.816485 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.817917 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.818604 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.818706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.818790 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.819310 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.819358 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
I20260619 10:25:16.819797 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.814624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41173 (local address 127.0.0.1:57778)
0619 10:25:16.814770 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.814826 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.814894 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.815205 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.815208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.815210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.815265 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.815266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.817920 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.817921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.818545 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.818546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.819568 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.819569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.819624 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.819627 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.819699 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.819734 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260619 10:25:16.819890 5130 tablet_service.cc:1467] Processing CreateTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 (DEFAULT_TABLE table=test-table [id=2f06f4604bca4a8d931abcf4292c3436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.820005 4868 tablet_service.cc:1467] Processing CreateTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 (DEFAULT_TABLE table=test-table [id=2f06f4604bca4a8d931abcf4292c3436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.820202 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.814773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.19.65.130:41173)
0619 10:25:16.814985 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.814990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.814999 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:16.815037 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.815039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.815053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.815126 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.815372 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.816660 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.818629 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.819393 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.819452 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.819534 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.820059 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.820116 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
I20260619 10:25:16.820257 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7faa6f6ca28452a825d2f21d152fed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.820289 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7faa6f6ca28452a825d2f21d152fed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.820822 4999 tablet_service.cc:1467] Processing CreateTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 (DEFAULT_TABLE table=test-table [id=2f06f4604bca4a8d931abcf4292c3436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:16.821117 4999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7faa6f6ca28452a825d2f21d152fed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:16.823418 5210 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: Bootstrap starting.
I20260619 10:25:16.823678 5212 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Bootstrap starting.
I20260619 10:25:16.824036 5210 tablet_bootstrap.cc:654] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.824268 5212 tablet_bootstrap.cc:654] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.824349 5210 log.cc:824] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.824584 5212 log.cc:824] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.824643 5211 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Bootstrap starting.
I20260619 10:25:16.825071 5210 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: No bootstrap required, opened a new log
I20260619 10:25:16.825145 5210 ts_tablet_manager.cc:1397] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:25:16.825268 5211 tablet_bootstrap.cc:654] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:16.825347 5212 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: No bootstrap required, opened a new log
I20260619 10:25:16.825412 5212 ts_tablet_manager.cc:1397] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:16.825606 5211 log.cc:824] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:16.826390 5211 tablet_bootstrap.cc:492] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: No bootstrap required, opened a new log
I20260619 10:25:16.826464 5211 ts_tablet_manager.cc:1397] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:16.827896 5210 raft_consensus.cc:348] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.828061 5210 raft_consensus.cc:374] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.828096 5210 raft_consensus.cc:729] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944d77b7927a47b8bb8171c61bb14e51, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.828052 5212 raft_consensus.cc:348] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.828174 5212 raft_consensus.cc:374] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.828230 5212 raft_consensus.cc:729] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37270f5aaaf4984aa1c158e51a349f2, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.828248 5210 consensus_queue.cc:260] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.828327 5212 consensus_queue.cc:260] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.828500 5210 ts_tablet_manager.cc:1428] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 10:25:16.828558 5212 ts_tablet_manager.cc:1428] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:16.828631 4934 heartbeater.cc:503] Master 127.19.65.190:42231 was elected leader, sending a full tablet report...
I20260619 10:25:16.828538 5211 raft_consensus.cc:348] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.828689 5065 heartbeater.cc:503] Master 127.19.65.190:42231 was elected leader, sending a full tablet report...
I20260619 10:25:16.828691 5211 raft_consensus.cc:374] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:16.828724 5211 raft_consensus.cc:729] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d0c7e826854afd9816bcff953a5ef0, State: Initialized, Role: FOLLOWER
I20260619 10:25:16.828825 5211 consensus_queue.cc:260] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.829066 5211 ts_tablet_manager.cc:1428] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:16.829177 5196 heartbeater.cc:503] Master 127.19.65.190:42231 was elected leader, sending a full tablet report...
I20260619 10:25:16.895449 5216 raft_consensus.cc:482] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:16.895557 5216 raft_consensus.cc:504] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.895856 5216 leader_election.cc:288] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a37270f5aaaf4984aa1c158e51a349f2 (127.19.65.130:41173), 71d0c7e826854afd9816bcff953a5ef0 (127.19.65.131:41409)
I20260619 10:25:16.898556 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.896113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:59311 (local address 127.19.65.130:41173)
0619 10:25:16.896166 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.896168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.896177 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:16.896208 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.896209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.896218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.896311 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.896473 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.896930 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.897854 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.898210 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.898295 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.898303 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.898444 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.898484 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 10:25:16.898556 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.895998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:41173 (local address 127.19.65.129:59311)
0619 10:25:16.896058 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.896071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.896129 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.896319 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.896321 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.896322 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.896394 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.896396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.897019 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.897020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.897757 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.897759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.898293 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.898294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.898369 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.898372 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.898439 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.898489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 10:25:16.898744 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.896189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:49781 (local address 127.19.65.131:41409)
0619 10:25:16.896235 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.896237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.896304 (+ 67us) server_negotiation.cc:407] Connection header received
0619 10:25:16.896383 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.896384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.896393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.896465 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:16.896624 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.897088 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.897997 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.898346 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.898426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.898433 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.898624 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.898667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:25:16.898882 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7faa6f6ca28452a825d2f21d152fed6" candidate_uuid: "944d77b7927a47b8bb8171c61bb14e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37270f5aaaf4984aa1c158e51a349f2" is_pre_election: true
I20260619 10:25:16.898957 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.896099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41409 (local address 127.19.65.129:49781)
0619 10:25:16.896244 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.896252 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:16.896314 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:16.896467 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:16.896468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:16.896470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:16.896550 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.896551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.897170 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.897171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.897905 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:16.897907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.898427 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.898428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:16.898515 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.898534 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 10:25:16.898835 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 10:25:16.898887 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260619 10:25:16.899056 5019 raft_consensus.cc:2454] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944d77b7927a47b8bb8171c61bb14e51 in term 0.
I20260619 10:25:16.899331 4823 leader_election.cc:302] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944d77b7927a47b8bb8171c61bb14e51, a37270f5aaaf4984aa1c158e51a349f2; no voters:
I20260619 10:25:16.899474 5216 raft_consensus.cc:2781] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:16.899469 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7faa6f6ca28452a825d2f21d152fed6" candidate_uuid: "944d77b7927a47b8bb8171c61bb14e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d0c7e826854afd9816bcff953a5ef0" is_pre_election: true
I20260619 10:25:16.899546 5216 raft_consensus.cc:482] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:16.899577 5216 raft_consensus.cc:3037] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.899611 5150 raft_consensus.cc:2454] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944d77b7927a47b8bb8171c61bb14e51 in term 0.
I20260619 10:25:16.900314 5216 raft_consensus.cc:504] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.900460 5216 leader_election.cc:288] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [CANDIDATE]: Term 1 election: Requested vote from peers a37270f5aaaf4984aa1c158e51a349f2 (127.19.65.130:41173), 71d0c7e826854afd9816bcff953a5ef0 (127.19.65.131:41409)
I20260619 10:25:16.900647 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7faa6f6ca28452a825d2f21d152fed6" candidate_uuid: "944d77b7927a47b8bb8171c61bb14e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37270f5aaaf4984aa1c158e51a349f2"
I20260619 10:25:16.900647 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7faa6f6ca28452a825d2f21d152fed6" candidate_uuid: "944d77b7927a47b8bb8171c61bb14e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d0c7e826854afd9816bcff953a5ef0"
I20260619 10:25:16.900729 5150 raft_consensus.cc:3037] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.900730 5019 raft_consensus.cc:3037] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:16.901619 5019 raft_consensus.cc:2454] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944d77b7927a47b8bb8171c61bb14e51 in term 1.
I20260619 10:25:16.901619 5150 raft_consensus.cc:2454] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944d77b7927a47b8bb8171c61bb14e51 in term 1.
I20260619 10:25:16.901788 4823 leader_election.cc:302] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944d77b7927a47b8bb8171c61bb14e51, a37270f5aaaf4984aa1c158e51a349f2; no voters:
I20260619 10:25:16.901886 5216 raft_consensus.cc:2781] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:16.902015 5216 raft_consensus.cc:686] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [term 1 LEADER]: Becoming Leader. State: Replica: 944d77b7927a47b8bb8171c61bb14e51, State: Running, Role: LEADER
I20260619 10:25:16.902083 5216 consensus_queue.cc:237] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } }
I20260619 10:25:16.902872 4746 catalog_manager.cc:5495] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 944d77b7927a47b8bb8171c61bb14e51 (127.19.65.129). New cstate: current_term: 1 leader_uuid: "944d77b7927a47b8bb8171c61bb14e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944d77b7927a47b8bb8171c61bb14e51" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 38253 } health_report { overall_health: HEALTHY } } }
I20260619 10:25:16.909336 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.907338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.19.65.129:38253)
0619 10:25:16.907375 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:16.907377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:16.907385 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:16.907426 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:16.907427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:16.907441 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:16.907511 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:16.907592 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.908021 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.908697 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:16.908863 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:16.908910 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:16.909032 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:16.909112 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:16.909168 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:16.909260 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 10:25:16.909274 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:16.910959 5019 raft_consensus.cc:1261] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 1 FOLLOWER]: Refusing update from remote peer 944d77b7927a47b8bb8171c61bb14e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:16.910987 5150 raft_consensus.cc:1261] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 1 FOLLOWER]: Refusing update from remote peer 944d77b7927a47b8bb8171c61bb14e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:16.911276 5220 consensus_queue.cc:1035] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a37270f5aaaf4984aa1c158e51a349f2" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:16.911403 5216 consensus_queue.cc:1035] T d7faa6f6ca28452a825d2f21d152fed6 P 944d77b7927a47b8bb8171c61bb14e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d0c7e826854afd9816bcff953a5ef0" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:16.913584 5223 mvcc.cc:205] Tried to move back new op lower bound from 7298517880464596992 to 7298517880431554560. Current Snapshot: MvccSnapshot[applied={T|T < 7298517880464596992}]
I20260619 10:25:16.914582 5224 mvcc.cc:205] Tried to move back new op lower bound from 7298517880464596992 to 7298517880431554560. Current Snapshot: MvccSnapshot[applied={T|T < 7298517880464596992}]
I20260619 10:25:16.915973 4746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:16.916066 4746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:16.916779 4746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b39f3bb25b549fbb7709fb5c6b17d8a: Sending DeleteTablet for 3 replicas of tablet d7faa6f6ca28452a825d2f21d152fed6
I20260619 10:25:16.917069 4999 tablet_service.cc:1514] Processing DeleteTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='slave'} at 127.0.0.1:57778
I20260619 10:25:16.917162 4868 tablet_service.cc:1514] Processing DeleteTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='slave'} at 127.0.0.1:52474
I20260619 10:25:16.917176 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet d7faa6f6ca28452a825d2f21d152fed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:16 UTC) from {username='slave'} at 127.0.0.1:57586
I20260619 10:25:16.917395 5234 tablet_replica.cc:330] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: stopping tablet replica
I20260619 10:25:16.917496 5234 raft_consensus.cc:2229] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:16.917599 5234 raft_consensus.cc:2258] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:16.917663 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4806
I20260619 10:25:16.917601 5232 tablet_replica.cc:330] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: stopping tablet replica
I20260619 10:25:16.918680 5232 raft_consensus.cc:2229] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:16.918839 5232 raft_consensus.cc:2258] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:16.918838 5234 ts_tablet_manager.cc:1905] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:16.919261 5232 ts_tablet_manager.cc:1905] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:16.920584 5234 ts_tablet_manager.cc:1918] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:16.920670 5234 log.cc:1196] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/d7faa6f6ca28452a825d2f21d152fed6
I20260619 10:25:16.920997 5234 ts_tablet_manager.cc:1939] T d7faa6f6ca28452a825d2f21d152fed6 P 71d0c7e826854afd9816bcff953a5ef0: Deleting consensus metadata
I20260619 10:25:16.921042 5232 ts_tablet_manager.cc:1918] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:16.921116 5232 log.cc:1196] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/d7faa6f6ca28452a825d2f21d152fed6
I20260619 10:25:16.921355 5232 ts_tablet_manager.cc:1939] T d7faa6f6ca28452a825d2f21d152fed6 P a37270f5aaaf4984aa1c158e51a349f2: Deleting consensus metadata
I20260619 10:25:16.921517 4732 catalog_manager.cc:4841] TS 71d0c7e826854afd9816bcff953a5ef0 (127.19.65.131:41409): tablet d7faa6f6ca28452a825d2f21d152fed6 (table test-table [id=2f06f4604bca4a8d931abcf4292c3436]) successfully deleted
I20260619 10:25:16.921716 4732 catalog_manager.cc:4841] TS a37270f5aaaf4984aa1c158e51a349f2 (127.19.65.130:41173): tablet d7faa6f6ca28452a825d2f21d152fed6 (table test-table [id=2f06f4604bca4a8d931abcf4292c3436]) successfully deleted
W20260619 10:25:16.923820 4732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:38253 (error 108)
I20260619 10:25:16.924029 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4937
I20260619 10:25:16.924639 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:16.924139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:38253 (local address 127.0.0.1:52490)
0619 10:25:16.924557 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:16.924603 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:38253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260619 10:25:16.925009 4732 catalog_manager.cc:4584] TS 944d77b7927a47b8bb8171c61bb14e51 (127.19.65.129:38253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7faa6f6ca28452a825d2f21d152fed6: Network error: Client connection negotiation failed: client connection to 127.19.65.129:38253: connect: Connection refused (error 111)
I20260619 10:25:16.928287 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5068
I20260619 10:25:16.932732 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 4715
2026-06-19T10:25:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (690 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:16 dist-test-slave-xd2q krb5kdc[5238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:16 dist-test-slave-xd2q krb5kdc[5238](info): set up 2 sockets
Jun 19 10:25:16 dist-test-slave-xd2q krb5kdc[5238](info): commencing operation
krb5kdc: starting...
W20260619 10:25:17.955010 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:18.063786 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46291
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:18.140590 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:18.140765 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:18.140786 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:18.141932 5251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:18.141973 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:18.141986 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:18.141999 5251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:18.142048 5251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:18.143523 5251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:18.143744 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:18.143976 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:18.148545 5256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.148563 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.148623 5251 server_base.cc:1034] running on GCE node
W20260619 10:25:18.148556 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.148941 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:18.149199 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:18.150370 5251 hybrid_clock.cc:648] HybridClock initialized: now 1781864718150343 us; error 39 us; skew 500 ppm
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:18.152711 5251 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:18.153075 5251 webserver.cc:460] Webserver started at http://127.19.65.190:40985/ using document root <none> and password file <none>
I20260619 10:25:18.153249 5251 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:18.153286 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:18.153370 5251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:18.154309 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "aca3e5201ebb4d7b820348cbd5ba2f24"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "aaf3161ca1959b4b04e40ed42601a92b"
server_key_iv: "a5dcf946c6ae7c1347f27abb2929b485"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.154601 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "aca3e5201ebb4d7b820348cbd5ba2f24"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "aaf3161ca1959b4b04e40ed42601a92b"
server_key_iv: "a5dcf946c6ae7c1347f27abb2929b485"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.155882 5251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:18.156487 5266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.156662 5251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:18.156740 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "aca3e5201ebb4d7b820348cbd5ba2f24"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "aaf3161ca1959b4b04e40ed42601a92b"
server_key_iv: "a5dcf946c6ae7c1347f27abb2929b485"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.156790 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:18.170228 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:18.172137 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:18.172236 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:18.175979 5251 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46291
I20260619 10:25:18.176021 5318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46291 every 8 connection(s)
I20260619 10:25:18.176374 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:18.177215 5319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:18.177500 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5251
I20260619 10:25:18.177626 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:18.177794 19718 external_mini_cluster.cc:1428] Setting key 80d93c368bbfb1612ece24fe0c2b8301
I20260619 10:25:18.180169 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: Bootstrap starting.
I20260619 10:25:18.180743 5319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:18.181025 5319 log.cc:824] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:18.181687 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: No bootstrap required, opened a new log
I20260619 10:25:18.183719 5319 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } }
I20260619 10:25:18.183889 5319 raft_consensus.cc:374] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:18.183959 5319 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3e5201ebb4d7b820348cbd5ba2f24, State: Initialized, Role: FOLLOWER
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:18.184124 5319 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } }
I20260619 10:25:18.184217 5319 raft_consensus.cc:388] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:18.184263 5319 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:18.184324 5319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:18.184959 5319 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } }
I20260619 10:25:18.185065 5319 leader_election.cc:302] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aca3e5201ebb4d7b820348cbd5ba2f24; no voters:
I20260619 10:25:18.185245 5319 leader_election.cc:288] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:18.185272 5324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:18.185389 5324 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [term 1 LEADER]: Becoming Leader. State: Replica: aca3e5201ebb4d7b820348cbd5ba2f24, State: Running, Role: LEADER
I20260619 10:25:18.185557 5319 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:18.185716 5324 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } }
I20260619 10:25:18.186065 5326 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } } }
I20260619 10:25:18.186075 5325 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca3e5201ebb4d7b820348cbd5ba2f24. Latest consensus state: current_term: 1 leader_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3e5201ebb4d7b820348cbd5ba2f24" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46291 } } }
I20260619 10:25:18.186179 5325 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:18.186198 5326 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:18.186868 5333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:18.187433 5333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:18.188660 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.178314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.19.65.190:46291)
0619 10:25:18.178488 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.178497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.178526 (+ 29us) server_negotiation.cc:407] Connection header received
0619 10:25:18.178913 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.178919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.178969 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.179080 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.179907 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.180774 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.181584 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.182121 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.182230 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.184691 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.184695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.184695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:18.184704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.186649 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.187108 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.187112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.187112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.187230 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.187563 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.187563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.187564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.187847 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.187999 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.188338 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.188465 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260619 10:25:18.189177 5333 catalog_manager.cc:1347] Generated new cluster ID: 7235320472674b749e792ecd895a979b
I20260619 10:25:18.189231 5333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:18.199784 5333 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:18.200238 5333 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:18.208736 5333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: Generated new TSK 0
I20260619 10:25:18.208909 5333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:25:18.236060 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:18.314486 5345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:18.314679 5345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:18.314702 5345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:18.315853 5345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:18.315910 5345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:18.315943 5345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:18.317339 5345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:18.317572 5345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:18.317797 5345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:18.322649 5353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.322649 5350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.322693 5351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.322800 5345 server_base.cc:1034] running on GCE node
I20260619 10:25:18.323082 5345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:18.323405 5345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:18.324554 5345 hybrid_clock.cc:648] HybridClock initialized: now 1781864718324537 us; error 35 us; skew 500 ppm
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:18.326956 5345 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:25:18.327435 5345 webserver.cc:460] Webserver started at http://127.19.65.129:36373/ using document root <none> and password file <none>
I20260619 10:25:18.327656 5345 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:18.327718 5345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:18.327843 5345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:18.328812 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "f8c453c5a60348e68c418367faf08113"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "88d3382974aff4ed1f07edc6b951dd6c"
server_key_iv: "77fa3ad5ee990472412c6f6f223cdc68"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.329144 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "f8c453c5a60348e68c418367faf08113"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "88d3382974aff4ed1f07edc6b951dd6c"
server_key_iv: "77fa3ad5ee990472412c6f6f223cdc68"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.330340 5345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:18.331000 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.331166 5345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:18.331254 5345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "f8c453c5a60348e68c418367faf08113"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "88d3382974aff4ed1f07edc6b951dd6c"
server_key_iv: "77fa3ad5ee990472412c6f6f223cdc68"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.331352 5345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:18.367609 5345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:18.369964 5345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:18.370116 5345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:18.370364 5345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:18.370657 5345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:18.370708 5345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.370743 5345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:18.370769 5345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.376374 5345 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:45579
I20260619 10:25:18.376446 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:45579 every 8 connection(s)
I20260619 10:25:18.376781 5345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:18.380504 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5345
I20260619 10:25:18.380662 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:18.380910 19718 external_mini_cluster.cc:1428] Setting key a2f912035e85dec7352dc7ec937bf746
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:18.386255 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.377498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:57057 (local address 127.19.65.190:46291)
0619 10:25:18.377545 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.377547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.377773 (+ 226us) server_negotiation.cc:407] Connection header received
0619 10:25:18.378205 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.378206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.378215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.378264 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.379139 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.379686 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.380616 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.380996 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.381088 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.383386 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.383387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.383388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:18.383394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.384766 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.385156 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.385157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.385158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.385217 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.385540 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.385541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.385542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.385650 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.385724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.386032 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.386128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:18.386427 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.377480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46291 (local address 127.19.65.129:57057)
0619 10:25:18.377654 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.377687 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.378121 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.378331 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.378334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.378865 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:18.379039 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.379043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.379758 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.379759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.380509 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.380511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.381103 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.381104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.381249 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.381747 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:18.381763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:18.383281 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:18.384865 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.384871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.384873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.385082 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.385283 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.385284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.385284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.385468 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.385716 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:18.385722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:18.385862 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.385999 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.386132 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260619 10:25:18.386860 5474 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46291
I20260619 10:25:18.386946 5474 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:18.387133 5474 heartbeater.cc:511] Master 127.19.65.190:46291 requested a full tablet report, sending...
I20260619 10:25:18.387674 5283 ts_manager.cc:194] Registered new tserver with Master: f8c453c5a60348e68c418367faf08113 (127.19.65.129:45579)
I20260619 10:25:18.388618 5283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:57057
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:25:18.404070 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:18.480208 5479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:18.480432 5479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:18.480466 5479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:18.481581 5479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:18.481647 5479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:18.481686 5479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:18.483047 5479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:18.483314 5479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:18.483536 5479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:18.488431 5487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.488431 5484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.488440 5485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.488582 5479 server_base.cc:1034] running on GCE node
I20260619 10:25:18.488812 5479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:18.489081 5479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:18.490257 5479 hybrid_clock.cc:648] HybridClock initialized: now 1781864718490245 us; error 28 us; skew 500 ppm
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:18.492729 5479 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:25:18.493137 5479 webserver.cc:460] Webserver started at http://127.19.65.130:37747/ using document root <none> and password file <none>
I20260619 10:25:18.493301 5479 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:18.493346 5479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:18.493428 5479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:18.494346 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "b41fcd3747e5421bb03afa15c7209e8b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "158866ed06b9d1cd923fd32fe496f54b"
server_key_iv: "0c5eb481ad0e7be6a27fb3dfefa6a3c6"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.494625 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "b41fcd3747e5421bb03afa15c7209e8b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "158866ed06b9d1cd923fd32fe496f54b"
server_key_iv: "0c5eb481ad0e7be6a27fb3dfefa6a3c6"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.495884 5479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:18.496513 5494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.496704 5479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:18.496788 5479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "b41fcd3747e5421bb03afa15c7209e8b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "158866ed06b9d1cd923fd32fe496f54b"
server_key_iv: "0c5eb481ad0e7be6a27fb3dfefa6a3c6"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.496834 5479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:18.509330 5479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:18.511492 5479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:18.511593 5479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:18.511758 5479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:18.512041 5479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:18.512071 5479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.512130 5479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:18.512157 5479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.517669 5479 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:37639
I20260619 10:25:18.517714 5607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:37639 every 8 connection(s)
I20260619 10:25:18.517980 5479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:18.518270 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5479
I20260619 10:25:18.518407 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:18.518666 19718 external_mini_cluster.cc:1428] Setting key 3fa24cc72c93fbe7b815f905cebcdf61
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:25:18.528146 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.518604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:48487 (local address 127.19.65.190:46291)
0619 10:25:18.518671 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.518674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.518904 (+ 230us) server_negotiation.cc:407] Connection header received
0619 10:25:18.519625 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.519626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.519635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.519692 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.520880 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.521651 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.522689 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.523034 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.523115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.525161 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.525163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.525163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:18.525171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.526716 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.527116 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.527117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.527117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.527182 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.527491 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.527492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.527492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.527605 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.527678 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.527975 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.528059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:25:18.528292 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.518590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46291 (local address 127.19.65.130:48487)
0619 10:25:18.518753 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.518809 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.519523 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.519757 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.519761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.520514 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:18.520769 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.520787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.521753 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.521754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.522594 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.522596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.523139 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.523140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.523243 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.523792 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:18.523800 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:18.525065 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:18.526803 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.526808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.526810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.527011 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.527258 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.527259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.527259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.527417 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.527668 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:18.527674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:18.527800 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.527939 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.528083 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260619 10:25:18.528781 5608 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46291
I20260619 10:25:18.528906 5608 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:18.529070 5608 heartbeater.cc:511] Master 127.19.65.190:46291 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
I20260619 10:25:18.529433 5283 ts_manager.cc:194] Registered new tserver with Master: b41fcd3747e5421bb03afa15c7209e8b (127.19.65.130:37639)
I20260619 10:25:18.529920 5283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:48487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:25:18.544075 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:18.621479 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:18.621687 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:18.621737 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:18.622797 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:18.622859 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:18.622916 5613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:18.624416 5613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:18.624682 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:18.624893 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:18.629910 5613 server_base.cc:1034] running on GCE node
W20260619 10:25:18.629848 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.629878 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.629884 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.630352 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:18.630591 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:18.631762 5613 hybrid_clock.cc:648] HybridClock initialized: now 1781864718631757 us; error 33 us; skew 500 ppm
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:18.634263 5613 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:25:18.634662 5613 webserver.cc:460] Webserver started at http://127.19.65.131:35809/ using document root <none> and password file <none>
I20260619 10:25:18.634888 5613 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:18.634976 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:18.635135 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:18.636162 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "2cabc608191845c7aaa18567e927beb8"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "5cb77c32494c13495ef306cc138045fe"
server_key_iv: "f29e86971d3e957c0264a8e7475841b5"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.636516 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "2cabc608191845c7aaa18567e927beb8"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "5cb77c32494c13495ef306cc138045fe"
server_key_iv: "f29e86971d3e957c0264a8e7475841b5"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.637746 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:18.638413 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.638599 5613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:18.638684 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "2cabc608191845c7aaa18567e927beb8"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "5cb77c32494c13495ef306cc138045fe"
server_key_iv: "f29e86971d3e957c0264a8e7475841b5"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.638746 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:18.652877 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:18.655042 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:18.655134 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:18.655386 5613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:18.655689 5613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:18.655719 5613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.655738 5613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:18.655750 5613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.661302 5613 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:44667
I20260619 10:25:18.661367 5741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:44667 every 8 connection(s)
I20260619 10:25:18.661656 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:25:18 dist-test-slave-xd2q krb5kdc[5238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:18.668227 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5613
I20260619 10:25:18.668347 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:18.668534 19718 external_mini_cluster.cc:1428] Setting key 769d56186366396374d92ce639aa6fd4
I20260619 10:25:18.670817 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.662289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:36333 (local address 127.19.65.190:46291)
0619 10:25:18.662365 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.662367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.662556 (+ 189us) server_negotiation.cc:407] Connection header received
0619 10:25:18.662993 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.662994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.663004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.663080 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.663941 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.664754 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.665866 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.666071 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.666154 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.668138 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.668139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.668139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:18.668145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.669673 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.670020 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.670021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.670021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.670112 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.670351 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.670352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.670352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.670459 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.670513 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.670708 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.670759 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:25:18.671046 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.662255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46291 (local address 127.19.65.131:36333)
0619 10:25:18.662411 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.662450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.662910 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.663129 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.663132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.663688 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:18.663863 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.663867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.664878 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.664879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.665759 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.665761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.666164 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.666165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.666289 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.666773 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:18.666780 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:18.668033 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:18.669736 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.669741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.669743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.669958 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.670193 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:18.670194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:18.670194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:18.670286 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:18.670519 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:18.670523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:18.670622 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.670696 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.670802 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260619 10:25:18.671486 5742 heartbeater.cc:347] Connected to a master server at 127.19.65.190:46291
I20260619 10:25:18.671571 5742 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:18.671716 5742 heartbeater.cc:511] Master 127.19.65.190:46291 requested a full tablet report, sending...
I20260619 10:25:18.672055 5283 ts_manager.cc:194] Registered new tserver with Master: 2cabc608191845c7aaa18567e927beb8 (127.19.65.131:44667)
I20260619 10:25:18.672421 5283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:36333
I20260619 10:25:18.680189 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:18.687510 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.681394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.19.65.190:46291)
0619 10:25:18.681424 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.681425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.681560 (+ 135us) server_negotiation.cc:407] Connection header received
0619 10:25:18.681625 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.681626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.681635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.681682 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.682188 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.682623 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.683385 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.683557 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.683612 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.684831 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.684831 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.684832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:18.684836 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.686347 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.686708 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.686709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.686709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.686751 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:18.686934 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:18.686935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:18.686935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:18.687130 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.687204 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.687411 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.687456 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:25:18.688985 5283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:18.690523 5283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:18.696681 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45579 (local address 127.0.0.1:59794)
0619 10:25:18.692640 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.692674 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.692729 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.693320 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.693321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.693324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:18.693421 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.693423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694424 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.694425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.695728 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.695731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696408 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.696488 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.696500 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.696568 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.696615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260619 10:25:18.696693 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.19.65.129:45579)
0619 10:25:18.693135 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.693139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.693148 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:18.693185 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.693191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.693222 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.693307 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:18.693525 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694319 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.695827 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696325 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696410 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.696465 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.696582 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.696622 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":94,"threads_started":1}
I20260619 10:25:18.696828 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:37639 (local address 127.0.0.1:45118)
0619 10:25:18.692822 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.692833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.692886 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.693260 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.693263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.693265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:18.693354 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.693356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694660 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.694661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.695738 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.695740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696635 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.696698 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.696701 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.696730 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.696766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":39,"threads_started":1}
I20260619 10:25:18.697099 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:44667 (local address 127.0.0.1:43418)
0619 10:25:18.693403 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.693414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.693463 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.693656 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.693657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.693659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:18.693709 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.693711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694860 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.694861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.695797 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.695799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696681 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.696742 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.696956 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.697004 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.697045 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":44,"threads_started":1}
I20260619 10:25:18.697145 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.19.65.130:37639)
0619 10:25:18.693095 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.693099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.693110 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:25:18.693152 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.693153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.693165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.693228 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:18.693449 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694558 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.695830 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696533 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696929 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.697002 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.697028 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.697066 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260619 10:25:18.697371 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.692922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.19.65.131:44667)
0619 10:25:18.693115 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.693119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.693534 (+ 415us) server_negotiation.cc:407] Connection header received
0619 10:25:18.693571 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.693573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.693586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.693659 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:18.693770 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.694769 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.695863 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.696533 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.696607 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.696654 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.697253 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.697294 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260619 10:25:18.697582 5408 tablet_service.cc:1467] Processing CreateTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 (DEFAULT_TABLE table=test-table [id=c797ca1f972b4203b1fb6f36074dbf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:18.697742 5542 tablet_service.cc:1467] Processing CreateTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 (DEFAULT_TABLE table=test-table [id=c797ca1f972b4203b1fb6f36074dbf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:18.697937 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f2a5de3c7438481ae0829c6b51a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:18.697891 5676 tablet_service.cc:1467] Processing CreateTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 (DEFAULT_TABLE table=test-table [id=c797ca1f972b4203b1fb6f36074dbf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:18.698053 5542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f2a5de3c7438481ae0829c6b51a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:18.698125 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f2a5de3c7438481ae0829c6b51a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:18.700836 5756 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: Bootstrap starting.
I20260619 10:25:18.701345 5758 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Bootstrap starting.
I20260619 10:25:18.701503 5756 tablet_bootstrap.cc:654] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:18.702013 5758 tablet_bootstrap.cc:654] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:18.702100 5756 log.cc:824] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:18.702585 5758 log.cc:824] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:18.702878 5756 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: No bootstrap required, opened a new log
I20260619 10:25:18.702908 5757 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Bootstrap starting.
I20260619 10:25:18.702955 5756 ts_tablet_manager.cc:1397] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:18.703543 5757 tablet_bootstrap.cc:654] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:18.703579 5758 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: No bootstrap required, opened a new log
I20260619 10:25:18.703670 5758 ts_tablet_manager.cc:1397] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:18.704089 5757 log.cc:824] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:18.704849 5757 tablet_bootstrap.cc:492] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: No bootstrap required, opened a new log
I20260619 10:25:18.704921 5757 ts_tablet_manager.cc:1397] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:18.705143 5756 raft_consensus.cc:348] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.705323 5756 raft_consensus.cc:374] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:18.705358 5756 raft_consensus.cc:729] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c453c5a60348e68c418367faf08113, State: Initialized, Role: FOLLOWER
I20260619 10:25:18.705488 5756 consensus_queue.cc:260] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.705466 5758 raft_consensus.cc:348] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.705634 5758 raft_consensus.cc:374] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:18.705675 5758 raft_consensus.cc:729] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41fcd3747e5421bb03afa15c7209e8b, State: Initialized, Role: FOLLOWER
I20260619 10:25:18.705754 5756 ts_tablet_manager.cc:1428] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:18.705792 5758 consensus_queue.cc:260] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.705895 5474 heartbeater.cc:503] Master 127.19.65.190:46291 was elected leader, sending a full tablet report...
I20260619 10:25:18.706056 5608 heartbeater.cc:503] Master 127.19.65.190:46291 was elected leader, sending a full tablet report...
I20260619 10:25:18.706058 5758 ts_tablet_manager.cc:1428] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:25:18.707345 5757 raft_consensus.cc:348] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.707510 5757 raft_consensus.cc:374] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:18.707531 5757 raft_consensus.cc:729] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cabc608191845c7aaa18567e927beb8, State: Initialized, Role: FOLLOWER
I20260619 10:25:18.707653 5757 consensus_queue.cc:260] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.707906 5742 heartbeater.cc:503] Master 127.19.65.190:46291 was elected leader, sending a full tablet report...
I20260619 10:25:18.707908 5757 ts_tablet_manager.cc:1428] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:18.733161 5763 raft_consensus.cc:482] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:18.733253 5763 raft_consensus.cc:504] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.733543 5763 leader_election.cc:288] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c453c5a60348e68c418367faf08113 (127.19.65.129:45579), 2cabc608191845c7aaa18567e927beb8 (127.19.65.131:44667)
I20260619 10:25:18.736332 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.733693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:57095 (local address 127.19.65.129:45579)
0619 10:25:18.733743 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.733745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.733780 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:25:18.733850 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.733851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.733863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.733928 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:18.734092 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.734600 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.735632 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.735986 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.736070 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.736079 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.736237 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.736271 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 10:25:18.736332 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.733660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45579 (local address 127.19.65.130:57095)
0619 10:25:18.733725 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.733737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.733795 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.733943 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.733944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.733946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:18.734011 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.734013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.734678 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.734679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.735514 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.735516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.736067 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.736068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.736151 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.736154 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.736228 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.736268 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 10:25:18.736439 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.733912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51083 (local address 127.19.65.131:44667)
0619 10:25:18.733955 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.733957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.733967 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:18.734045 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.734045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.734053 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.734101 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:18.734239 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.734761 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.735746 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.736106 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.736186 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.736193 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.736332 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.736367 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:25:18.736447 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.733769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:44667 (local address 127.19.65.130:51083)
0619 10:25:18.733923 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.733932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:18.733979 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:18.734102 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:18.734103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:18.734104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:18.734173 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.734175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.734870 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.734871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.735664 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:18.735666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.736186 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.736187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:18.736253 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.736256 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:18.736319 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 10:25:18.736355 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260619 10:25:18.736678 5696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f2a5de3c7438481ae0829c6b51a88" candidate_uuid: "b41fcd3747e5421bb03afa15c7209e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cabc608191845c7aaa18567e927beb8" is_pre_election: true
I20260619 10:25:18.736814 5696 raft_consensus.cc:2454] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41fcd3747e5421bb03afa15c7209e8b in term 0.
I20260619 10:25:18.736807 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f2a5de3c7438481ae0829c6b51a88" candidate_uuid: "b41fcd3747e5421bb03afa15c7209e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c453c5a60348e68c418367faf08113" is_pre_election: true
I20260619 10:25:18.736919 5428 raft_consensus.cc:2454] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41fcd3747e5421bb03afa15c7209e8b in term 0.
I20260619 10:25:18.737036 5496 leader_election.cc:302] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cabc608191845c7aaa18567e927beb8, b41fcd3747e5421bb03afa15c7209e8b; no voters:
I20260619 10:25:18.737162 5763 raft_consensus.cc:2781] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:18.737224 5763 raft_consensus.cc:482] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:18.737258 5763 raft_consensus.cc:3037] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:18.738009 5763 raft_consensus.cc:504] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.738135 5763 leader_election.cc:288] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [CANDIDATE]: Term 1 election: Requested vote from peers f8c453c5a60348e68c418367faf08113 (127.19.65.129:45579), 2cabc608191845c7aaa18567e927beb8 (127.19.65.131:44667)
I20260619 10:25:18.738281 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f2a5de3c7438481ae0829c6b51a88" candidate_uuid: "b41fcd3747e5421bb03afa15c7209e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c453c5a60348e68c418367faf08113"
I20260619 10:25:18.738299 5696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f2a5de3c7438481ae0829c6b51a88" candidate_uuid: "b41fcd3747e5421bb03afa15c7209e8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cabc608191845c7aaa18567e927beb8"
I20260619 10:25:18.738356 5428 raft_consensus.cc:3037] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:18.738364 5696 raft_consensus.cc:3037] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:18.739040 5428 raft_consensus.cc:2454] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41fcd3747e5421bb03afa15c7209e8b in term 1.
I20260619 10:25:18.739044 5696 raft_consensus.cc:2454] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41fcd3747e5421bb03afa15c7209e8b in term 1.
I20260619 10:25:18.739217 5496 leader_election.cc:302] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cabc608191845c7aaa18567e927beb8, b41fcd3747e5421bb03afa15c7209e8b; no voters:
I20260619 10:25:18.739347 5763 raft_consensus.cc:2781] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:18.739494 5763 raft_consensus.cc:686] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 1 LEADER]: Becoming Leader. State: Replica: b41fcd3747e5421bb03afa15c7209e8b, State: Running, Role: LEADER
I20260619 10:25:18.739589 5763 consensus_queue.cc:237] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } }
I20260619 10:25:18.740356 5282 catalog_manager.cc:5495] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b reported cstate change: term changed from 0 to 1, leader changed from <none> to b41fcd3747e5421bb03afa15c7209e8b (127.19.65.130). New cstate: current_term: 1 leader_uuid: "b41fcd3747e5421bb03afa15c7209e8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41fcd3747e5421bb03afa15c7209e8b" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 } health_report { overall_health: UNKNOWN } } }
I20260619 10:25:18.753211 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.751371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.19.65.130:37639)
0619 10:25:18.751406 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.751408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.751440 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:18.751485 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.751485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.751498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.751549 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:18.751653 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.752042 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.752615 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.752781 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.752822 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.752947 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:18.753015 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:18.753101 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.753141 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.753149 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:18.754796 5428 raft_consensus.cc:1261] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 1 FOLLOWER]: Refusing update from remote peer b41fcd3747e5421bb03afa15c7209e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:18.754796 5696 raft_consensus.cc:1261] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 1 FOLLOWER]: Refusing update from remote peer b41fcd3747e5421bb03afa15c7209e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:18.755117 5766 consensus_queue.cc:1035] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cabc608191845c7aaa18567e927beb8" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:18.755239 5763 consensus_queue.cc:1035] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c453c5a60348e68c418367faf08113" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:18.757565 5771 mvcc.cc:205] Tried to move back new op lower bound from 7298517888017252352 to 7298517887957962752. Current Snapshot: MvccSnapshot[applied={T|T < 7298517888017252352}]
I20260619 10:25:18.759616 5283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:18.759696 5283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:18.760300 5283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aca3e5201ebb4d7b820348cbd5ba2f24: Sending DeleteTablet for 3 replicas of tablet a87f2a5de3c7438481ae0829c6b51a88
I20260619 10:25:18.760603 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:59794
I20260619 10:25:18.760618 5542 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:45118
I20260619 10:25:18.760649 5676 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f2a5de3c7438481ae0829c6b51a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:18 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43418
I20260619 10:25:18.760880 5779 tablet_replica.cc:330] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113: stopping tablet replica
I20260619 10:25:18.760974 5779 raft_consensus.cc:2229] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:18.761030 5781 tablet_replica.cc:330] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: stopping tablet replica
I20260619 10:25:18.761068 5779 raft_consensus.cc:2258] T a87f2a5de3c7438481ae0829c6b51a88 P f8c453c5a60348e68c418367faf08113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:18.761147 5781 raft_consensus.cc:2229] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:18.761194 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5345
I20260619 10:25:18.761278 5781 raft_consensus.cc:2258] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:18.761699 5781 ts_tablet_manager.cc:1905] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:18.763577 5780 tablet_replica.cc:330] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: stopping tablet replica
I20260619 10:25:18.763692 5780 raft_consensus.cc:2229] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:18.763793 5780 raft_consensus.cc:2258] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:18.764219 5780 ts_tablet_manager.cc:1905] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:18.765516 5780 ts_tablet_manager.cc:1918] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:18.765605 5780 log.cc:1196] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/a87f2a5de3c7438481ae0829c6b51a88
I20260619 10:25:18.765853 5780 ts_tablet_manager.cc:1939] T a87f2a5de3c7438481ae0829c6b51a88 P b41fcd3747e5421bb03afa15c7209e8b: Deleting consensus metadata
I20260619 10:25:18.766207 5268 catalog_manager.cc:4841] TS b41fcd3747e5421bb03afa15c7209e8b (127.19.65.130:37639): tablet a87f2a5de3c7438481ae0829c6b51a88 (table test-table [id=c797ca1f972b4203b1fb6f36074dbf11]) successfully deleted
I20260619 10:25:18.767421 5781 ts_tablet_manager.cc:1918] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:18.767499 5781 log.cc:1196] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/a87f2a5de3c7438481ae0829c6b51a88
I20260619 10:25:18.767735 5781 ts_tablet_manager.cc:1939] T a87f2a5de3c7438481ae0829c6b51a88 P 2cabc608191845c7aaa18567e927beb8: Deleting consensus metadata
I20260619 10:25:18.768090 5268 catalog_manager.cc:4841] TS 2cabc608191845c7aaa18567e927beb8 (127.19.65.131:44667): tablet a87f2a5de3c7438481ae0829c6b51a88 (table test-table [id=c797ca1f972b4203b1fb6f36074dbf11]) successfully deleted
W20260619 10:25:18.768303 5268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:45579 (error 108)
I20260619 10:25:18.768563 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5479
I20260619 10:25:18.771688 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.771543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45579 (local address 127.0.0.1:59802)
0619 10:25:18.771621 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:18.771662 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 10:25:18.771827 5268 catalog_manager.cc:4584] TS f8c453c5a60348e68c418367faf08113 (127.19.65.129:45579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a87f2a5de3c7438481ae0829c6b51a88: Network error: Client connection negotiation failed: client connection to 127.19.65.129:45579: connect: Connection refused (error 111)
I20260619 10:25:18.773968 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5613
I20260619 10:25:18.779516 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5251
2026-06-19T10:25:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1848 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T10:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:18Z Disabled control of system clock
I20260619 10:25:18.806964 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41957
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:18.887351 5786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:18.887547 5786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:18.887573 5786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:18.888625 5786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:18.888657 5786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:18.888669 5786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:18.888681 5786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:18.888693 5786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:18.889959 5786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:41957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:18.890137 5786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:18.890358 5786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:18.894898 5794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.895000 5786 server_base.cc:1034] running on GCE node
W20260619 10:25:18.894912 5791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:18.894914 5792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:18.895443 5786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:18.895761 5786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:18.896927 5786 hybrid_clock.cc:648] HybridClock initialized: now 1781864718896922 us; error 38 us; skew 500 ppm
I20260619 10:25:18.898093 5786 webserver.cc:460] Webserver started at http://127.19.65.190:37549/ using document root <none> and password file <none>
I20260619 10:25:18.898365 5786 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:18.898435 5786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:18.898583 5786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:18.899706 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "14109d25abd740018fb0142e0b1ee22b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "631c9e155f77079f0a05960d6e613408"
server_key_iv: "cf9b6997a2f2f0f01ed83daf1e6524da"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.900117 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "14109d25abd740018fb0142e0b1ee22b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "631c9e155f77079f0a05960d6e613408"
server_key_iv: "cf9b6997a2f2f0f01ed83daf1e6524da"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.901481 5786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:18.902233 5800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:18.902534 5786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:18.902637 5786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "14109d25abd740018fb0142e0b1ee22b"
format_stamp: "Formatted at 2026-06-19 10:25:18 on dist-test-slave-xd2q"
server_key: "631c9e155f77079f0a05960d6e613408"
server_key_iv: "cf9b6997a2f2f0f01ed83daf1e6524da"
server_key_version: "encryptionkey@0"
I20260619 10:25:18.902714 5786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:18.909768 5786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:18.910049 5786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:18.910184 5786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:18.914053 5786 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:41957
I20260619 10:25:18.914081 5852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:41957 every 8 connection(s)
I20260619 10:25:18.914412 5786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:18.915365 5853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:18.917526 5853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: Bootstrap starting.
I20260619 10:25:18.918035 5853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:18.918265 5853 log.cc:824] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:18.918838 5853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: No bootstrap required, opened a new log
I20260619 10:25:18.920429 5853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } }
I20260619 10:25:18.920562 5853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:18.920584 5853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14109d25abd740018fb0142e0b1ee22b, State: Initialized, Role: FOLLOWER
I20260619 10:25:18.920678 5853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } }
I20260619 10:25:18.920728 5853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:18.920748 5853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:18.920815 5853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:18.921021 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5786
I20260619 10:25:18.921137 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:18.921331 19718 external_mini_cluster.cc:1428] Setting key 4936b43f755d2db5202fbc27444b1e22
I20260619 10:25:18.921425 5853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } }
I20260619 10:25:18.921545 5853 leader_election.cc:302] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14109d25abd740018fb0142e0b1ee22b; no voters:
I20260619 10:25:18.921705 5853 leader_election.cc:288] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:18.921793 5856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:18.921967 5856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [term 1 LEADER]: Becoming Leader. State: Replica: 14109d25abd740018fb0142e0b1ee22b, State: Running, Role: LEADER
I20260619 10:25:18.922155 5856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } }
I20260619 10:25:18.921972 5853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:18.923051 5860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14109d25abd740018fb0142e0b1ee22b. Latest consensus state: current_term: 1 leader_uuid: "14109d25abd740018fb0142e0b1ee22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } } }
I20260619 10:25:18.923151 5860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:18.923349 5857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14109d25abd740018fb0142e0b1ee22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14109d25abd740018fb0142e0b1ee22b" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 41957 } } }
I20260619 10:25:18.923415 5857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:18.924052 5873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:18.924136 5873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:18.924203 5867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:18.924371 5867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:18.925902 5867 catalog_manager.cc:1347] Generated new cluster ID: ec87e48606b341ecb5ae3bb8c1b901ee
I20260619 10:25:18.925953 5867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:18.927798 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:18.921831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.19.65.190:41957)
0619 10:25:18.922122 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:18.922131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:18.922177 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:25:18.922300 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:18.922307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:18.922375 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:18.922511 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:18.922749 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.924757 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.925843 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:18.926313 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:18.926446 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:18.927112 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:18.927115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:18.927115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:18.927282 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:18.927361 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:18.927376 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:18.927453 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:18.927526 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:25:18.927601 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260619 10:25:18.940394 5867 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:18.940881 5867 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:18.943987 5867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: Generated new TSK 0
I20260619 10:25:18.944152 5867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:18.947961 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--builtin_ntp_servers=127.19.65.148:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:19.038390 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:19.038578 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:19.038599 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:19.039724 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:19.039759 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:19.039784 5877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:19.041057 5877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:19.041270 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:19.041566 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:19.046967 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.047010 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.047118 5885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:19.047389 5877 server_base.cc:1034] running on GCE node
I20260619 10:25:19.047601 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:19.047827 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:19.048983 5877 hybrid_clock.cc:648] HybridClock initialized: now 1781864719048954 us; error 46 us; skew 500 ppm
I20260619 10:25:19.050139 5877 webserver.cc:460] Webserver started at http://127.19.65.129:36173/ using document root <none> and password file <none>
I20260619 10:25:19.050366 5877 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:19.050446 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:19.050597 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:19.051635 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "0f064faa82ea482da6108bb271f6a392"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "a25dc29e91a1e50f4da1d08620182807"
server_key_iv: "6e0bd505b2733f839f7327e8265aa94a"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.051983 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "0f064faa82ea482da6108bb271f6a392"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "a25dc29e91a1e50f4da1d08620182807"
server_key_iv: "6e0bd505b2733f839f7327e8265aa94a"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.053273 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:19.054039 5891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.054219 5877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:19.054314 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "0f064faa82ea482da6108bb271f6a392"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "a25dc29e91a1e50f4da1d08620182807"
server_key_iv: "6e0bd505b2733f839f7327e8265aa94a"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.054395 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:19.067273 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:19.067605 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:19.067751 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:19.067957 5877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:19.068254 5877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:19.068305 5877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.068337 5877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:19.068363 5877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.074155 5877 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:40669
I20260619 10:25:19.074196 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:40669 every 8 connection(s)
I20260619 10:25:19.074497 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:19.079610 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.075142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:32933 (local address 127.19.65.190:41957)
0619 10:25:19.075184 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.075186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.075440 (+ 254us) server_negotiation.cc:407] Connection header received
0619 10:25:19.075992 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.075993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.076008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.076070 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:19.076712 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.077211 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.078173 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.078379 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.078455 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.079325 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:19.079327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:19.079327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:19.079332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:19.079347 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:19.079352 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:19.079405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.079503 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.079536 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:19.079850 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.075129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41957 (local address 127.19.65.129:32933)
0619 10:25:19.075276 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.075347 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.075865 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.076136 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.076140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.076148 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:19.076615 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.076619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.077317 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.077318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.078079 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.078081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.078464 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.078465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.078591 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.079175 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:19.079176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:19.079202 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:19.079220 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:19.079407 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:19.079412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:19.079415 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.079506 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.079599 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260619 10:25:19.080401 6005 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41957
I20260619 10:25:19.080475 6005 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:19.080654 6005 heartbeater.cc:511] Master 127.19.65.190:41957 requested a full tablet report, sending...
I20260619 10:25:19.081130 5817 ts_manager.cc:194] Registered new tserver with Master: 0f064faa82ea482da6108bb271f6a392 (127.19.65.129:40669)
I20260619 10:25:19.081701 5817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.129:32933
I20260619 10:25:19.083194 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 5877
I20260619 10:25:19.083279 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:19.083515 19718 external_mini_cluster.cc:1428] Setting key 8877e8b4bb8bcf25678bfaac0a32022d
I20260619 10:25:19.084393 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--builtin_ntp_servers=127.19.65.148:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:19.161396 6008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:19.161587 6008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:19.161626 6008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:19.162648 6008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:19.162679 6008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:19.162703 6008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:19.163966 6008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:19.164132 6008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:19.164390 6008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:19.169255 6014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.169250 6016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.169343 6013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:19.169625 6008 server_base.cc:1034] running on GCE node
I20260619 10:25:19.169763 6008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:19.169965 6008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:19.171075 6008 hybrid_clock.cc:648] HybridClock initialized: now 1781864719171058 us; error 28 us; skew 500 ppm
I20260619 10:25:19.172288 6008 webserver.cc:460] Webserver started at http://127.19.65.130:45757/ using document root <none> and password file <none>
I20260619 10:25:19.172515 6008 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:19.172559 6008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:19.172654 6008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:19.173630 6008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "4e35e0569a0546be9bcca81bb84a8350"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "7064ee75ae28405f19fd1457ac938198"
server_key_iv: "67977c3d54104f371b9146abd2672c7d"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.173921 6008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "4e35e0569a0546be9bcca81bb84a8350"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "7064ee75ae28405f19fd1457ac938198"
server_key_iv: "67977c3d54104f371b9146abd2672c7d"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.175063 6008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:19.175742 6022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.175884 6008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.175966 6008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "4e35e0569a0546be9bcca81bb84a8350"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "7064ee75ae28405f19fd1457ac938198"
server_key_iv: "67977c3d54104f371b9146abd2672c7d"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.176017 6008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:19.197048 6008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:19.197286 6008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:19.197386 6008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:19.197587 6008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:19.197926 6008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:19.197960 6008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.197981 6008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:19.198055 6008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.203579 6008 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:38587
I20260619 10:25:19.203658 6135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:38587 every 8 connection(s)
I20260619 10:25:19.203969 6008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:19.208690 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6008
I20260619 10:25:19.208811 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:19.209020 19718 external_mini_cluster.cc:1428] Setting key 5a4ec45f84026a7533d73e7d86b9abb2
I20260619 10:25:19.209431 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.204698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34013 (local address 127.19.65.190:41957)
0619 10:25:19.204764 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.204767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.204981 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:25:19.205564 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.205565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.205576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.205651 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:19.206331 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.206933 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.207941 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.208156 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.208233 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.209128 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:19.209130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:19.209130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:19.209135 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:19.209151 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:19.209155 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:19.209221 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.209321 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.209355 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:25:19.209631 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.204691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41957 (local address 127.19.65.130:34013)
0619 10:25:19.204861 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.204894 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.205461 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.205733 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.205736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.205745 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:19.206228 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.206233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.207029 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.207030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.207833 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.207836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.208240 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.208241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.208378 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.208959 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:19.208960 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:19.208988 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:19.209009 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:19.209237 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:19.209244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:19.209247 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.209319 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.209426 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260619 10:25:19.210058 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--builtin_ntp_servers=127.19.65.148:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:25:19.211185 6136 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41957
I20260619 10:25:19.211262 6136 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:19.211500 6136 heartbeater.cc:511] Master 127.19.65.190:41957 requested a full tablet report, sending...
I20260619 10:25:19.211896 5817 ts_manager.cc:194] Registered new tserver with Master: 4e35e0569a0546be9bcca81bb84a8350 (127.19.65.130:38587)
I20260619 10:25:19.212321 5817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.130:34013
W20260619 10:25:19.289361 6139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:19.289551 6139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:19.289572 6139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:19.290707 6139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:19.290746 6139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:19.290771 6139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:19.292165 6139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:19.292418 6139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:19.292703 6139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:19.297726 6144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.297711 6145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:19.297701 6147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:19.297940 6139 server_base.cc:1034] running on GCE node
I20260619 10:25:19.298111 6139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:19.298302 6139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:19.299422 6139 hybrid_clock.cc:648] HybridClock initialized: now 1781864719299408 us; error 27 us; skew 500 ppm
I20260619 10:25:19.300675 6139 webserver.cc:460] Webserver started at http://127.19.65.131:35171/ using document root <none> and password file <none>
I20260619 10:25:19.301013 6139 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:19.301091 6139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:19.301297 6139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:19.302423 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "d2838ef5e1aa44009d9384e5d04b6341"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "4d5c1b64539e6b2a0745793bccb5aa4d"
server_key_iv: "8c1e045ad083a6124f0de06f2022cf1f"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.302790 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "d2838ef5e1aa44009d9384e5d04b6341"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "4d5c1b64539e6b2a0745793bccb5aa4d"
server_key_iv: "8c1e045ad083a6124f0de06f2022cf1f"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.304176 6139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:19.304963 6153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.305131 6139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:19.305241 6139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "d2838ef5e1aa44009d9384e5d04b6341"
format_stamp: "Formatted at 2026-06-19 10:25:19 on dist-test-slave-xd2q"
server_key: "4d5c1b64539e6b2a0745793bccb5aa4d"
server_key_iv: "8c1e045ad083a6124f0de06f2022cf1f"
server_key_version: "encryptionkey@0"
I20260619 10:25:19.305318 6139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:19.322489 6139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:19.322830 6139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:19.322997 6139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:19.323263 6139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:19.323681 6139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:19.323738 6139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.323773 6139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:19.323807 6139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:19.329751 6139 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:41191
I20260619 10:25:19.329792 6266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:41191 every 8 connection(s)
I20260619 10:25:19.330154 6139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:19.334892 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6139
I20260619 10:25:19.335013 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:19.335253 19718 external_mini_cluster.cc:1428] Setting key 6776314e79b441002d6f5311e69f8067
I20260619 10:25:19.335572 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.330913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:59577 (local address 127.19.65.190:41957)
0619 10:25:19.330992 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.330994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.331202 (+ 208us) server_negotiation.cc:407] Connection header received
0619 10:25:19.331789 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.331790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.331800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.331855 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:19.332485 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.332961 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.333993 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.334180 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.334232 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.335190 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:19.335192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:19.335192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:19.335198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:19.335216 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:19.335221 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:19.335303 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.335448 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.335483 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 10:25:19.335798 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.330914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:41957 (local address 127.19.65.131:59577)
0619 10:25:19.331095 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.331127 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.331684 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.331907 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.331910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.331918 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:19.332416 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.332420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.333049 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.333049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.333913 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.333915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.334266 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.334267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.334390 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.335026 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:19.335026 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:19.335065 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:19.335092 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:19.335323 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:19.335327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:19.335330 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.335410 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.335522 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260619 10:25:19.336295 6267 heartbeater.cc:347] Connected to a master server at 127.19.65.190:41957
I20260619 10:25:19.336423 6267 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:19.336587 6267 heartbeater.cc:511] Master 127.19.65.190:41957 requested a full tablet report, sending...
I20260619 10:25:19.336939 5817 ts_manager.cc:194] Registered new tserver with Master: d2838ef5e1aa44009d9384e5d04b6341 (127.19.65.131:41191)
I20260619 10:25:19.337288 5817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.131:59577
I20260619 10:25:19.347457 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:19.351413 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.348806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.19.65.190:41957)
0619 10:25:19.348858 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.348860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.348909 (+ 49us) server_negotiation.cc:407] Connection header received
0619 10:25:19.348959 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.348959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.348970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.349017 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:19.349168 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.349609 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.350269 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.350442 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.350489 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.351136 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:19.351137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:19.351137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:19.351141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:19.351150 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:19.351153 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:19.351210 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.351278 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.351334 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:25:19.352967 5817 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:19.354501 5817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:19.360611 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.19.65.130:38587)
0619 10:25:19.356876 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.356880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.357067 (+ 187us) server_negotiation.cc:407] Connection header received
0619 10:25:19.357115 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.357117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.357132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.357205 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:19.357403 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358521 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.359764 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360220 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360312 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360407 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.360482 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.360518 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260619 10:25:19.360611 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:38587 (local address 127.0.0.1:47590)
0619 10:25:19.356991 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.356999 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.357046 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.357269 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.357270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.357272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:19.357328 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.357330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358603 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.358604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.359686 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.359688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360315 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.360386 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360398 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.360470 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.360523 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":9,"thread_start_us":78,"threads_started":1}
I20260619 10:25:19.360761 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.19.65.129:40669)
0619 10:25:19.357005 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.357008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.357234 (+ 226us) server_negotiation.cc:407] Connection header received
0619 10:25:19.357279 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.357281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.357296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.357364 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:19.357529 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358577 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.359599 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360284 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360366 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360450 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.360646 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.360686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260619 10:25:19.360616 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41191 (local address 127.0.0.1:51974)
0619 10:25:19.356771 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.356808 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.356885 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.357164 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.357167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.357169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:19.357247 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.357249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358335 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.358336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.359569 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.359572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360387 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.360446 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360449 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.360479 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.360524 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260619 10:25:19.360893 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.19.65.131:41191)
0619 10:25:19.356971 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.356975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.356986 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:25:19.357024 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.357026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.357040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.357136 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:19.357328 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358239 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.359636 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360287 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360655 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360738 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.360762 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.360802 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":85,"threads_started":1}
I20260619 10:25:19.360939 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.356707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40669 (local address 127.0.0.1:53838)
0619 10:25:19.357130 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.357138 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.357180 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.357426 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.357427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.357429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:19.357460 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.357461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.358656 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.358657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.359523 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.359526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.360480 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.360481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.360527 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.360530 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.360574 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.360606 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":71,"threads_started":1}
I20260619 10:25:19.361552 6201 tablet_service.cc:1467] Processing CreateTablet for tablet b5044709797a4cbb9ba31e82e1da073e (DEFAULT_TABLE table=test-table [id=8889c1225fbd43b496dfc9d26ada7c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:19.361708 5939 tablet_service.cc:1467] Processing CreateTablet for tablet b5044709797a4cbb9ba31e82e1da073e (DEFAULT_TABLE table=test-table [id=8889c1225fbd43b496dfc9d26ada7c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:19.361927 6201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5044709797a4cbb9ba31e82e1da073e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:19.362040 5939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5044709797a4cbb9ba31e82e1da073e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:19.362035 6070 tablet_service.cc:1467] Processing CreateTablet for tablet b5044709797a4cbb9ba31e82e1da073e (DEFAULT_TABLE table=test-table [id=8889c1225fbd43b496dfc9d26ada7c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:19.362306 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5044709797a4cbb9ba31e82e1da073e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:19.365334 6283 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Bootstrap starting.
I20260619 10:25:19.365347 6282 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Bootstrap starting.
I20260619 10:25:19.365487 6281 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Bootstrap starting.
I20260619 10:25:19.366012 6282 tablet_bootstrap.cc:654] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:19.366015 6283 tablet_bootstrap.cc:654] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:19.366133 6281 tablet_bootstrap.cc:654] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:19.366329 6283 log.cc:824] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:19.366338 6282 log.cc:824] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:19.366446 6281 log.cc:824] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:19.367034 6282 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: No bootstrap required, opened a new log
I20260619 10:25:19.367086 6283 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: No bootstrap required, opened a new log
I20260619 10:25:19.367117 6282 ts_tablet_manager.cc:1397] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:19.367149 6281 tablet_bootstrap.cc:492] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: No bootstrap required, opened a new log
I20260619 10:25:19.367157 6283 ts_tablet_manager.cc:1397] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:19.367206 6281 ts_tablet_manager.cc:1397] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:19.368983 6282 raft_consensus.cc:348] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.369156 6282 raft_consensus.cc:374] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:19.369197 6282 raft_consensus.cc:729] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2838ef5e1aa44009d9384e5d04b6341, State: Initialized, Role: FOLLOWER
I20260619 10:25:19.369302 6282 consensus_queue.cc:260] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.369532 6282 ts_tablet_manager.cc:1428] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 10:25:19.369750 6267 heartbeater.cc:503] Master 127.19.65.190:41957 was elected leader, sending a full tablet report...
I20260619 10:25:19.369668 6283 raft_consensus.cc:348] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.369668 6281 raft_consensus.cc:348] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.369804 6283 raft_consensus.cc:374] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:19.369804 6281 raft_consensus.cc:374] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:19.369838 6283 raft_consensus.cc:729] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e35e0569a0546be9bcca81bb84a8350, State: Initialized, Role: FOLLOWER
I20260619 10:25:19.369850 6281 raft_consensus.cc:729] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f064faa82ea482da6108bb271f6a392, State: Initialized, Role: FOLLOWER
I20260619 10:25:19.369951 6281 consensus_queue.cc:260] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.369982 6283 consensus_queue.cc:260] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.370220 6283 ts_tablet_manager.cc:1428] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:19.370220 6281 ts_tablet_manager.cc:1428] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:25:19.370252 6005 heartbeater.cc:503] Master 127.19.65.190:41957 was elected leader, sending a full tablet report...
I20260619 10:25:19.370342 6136 heartbeater.cc:503] Master 127.19.65.190:41957 was elected leader, sending a full tablet report...
W20260619 10:25:19.454520 6137 tablet.cc:2369] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:25:19.575229 6006 tablet.cc:2369] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:25:19.580727 6268 tablet.cc:2369] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:19.645109 6289 raft_consensus.cc:482] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:19.645217 6289 raft_consensus.cc:504] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.645509 6289 leader_election.cc:288] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2838ef5e1aa44009d9384e5d04b6341 (127.19.65.131:41191), 0f064faa82ea482da6108bb271f6a392 (127.19.65.129:40669)
I20260619 10:25:19.649191 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:45455 (local address 127.19.65.131:41191)
0619 10:25:19.645869 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.645871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.645880 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:19.645912 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.645913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.645922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.645987 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:19.646147 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.646851 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648296 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.648835 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648915 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.648923 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.649085 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.649127 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:25:19.649212 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.645865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:51761 (local address 127.19.65.129:40669)
0619 10:25:19.645906 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.645908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.646100 (+ 192us) server_negotiation.cc:407] Connection header received
0619 10:25:19.646147 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.646149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.646159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.646210 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:19.646354 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.646981 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648296 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.648856 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648917 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.648923 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.649102 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.649156 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:25:19.649245 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.645699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:41191 (local address 127.19.65.130:45455)
0619 10:25:19.645772 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.645784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.645838 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.645995 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.645996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.645998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:19.646068 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.646070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.646939 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.646940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.648200 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.648203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.648925 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.649003 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.649007 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.649065 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.649145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3}
I20260619 10:25:19.649250 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.645843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40669 (local address 127.19.65.130:51761)
0619 10:25:19.646018 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.646029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:19.646085 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:19.646225 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:19.646226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:19.646227 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:19.646272 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.646273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.647043 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.647043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.648209 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:19.648210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.648936 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.648936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:19.649004 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.649007 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:19.649066 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:25:19.649150 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":61,"thread_start_us":96,"threads_started":1}
I20260619 10:25:19.649621 6221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5044709797a4cbb9ba31e82e1da073e" candidate_uuid: "4e35e0569a0546be9bcca81bb84a8350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2838ef5e1aa44009d9384e5d04b6341" is_pre_election: true
I20260619 10:25:19.649688 5959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5044709797a4cbb9ba31e82e1da073e" candidate_uuid: "4e35e0569a0546be9bcca81bb84a8350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f064faa82ea482da6108bb271f6a392" is_pre_election: true
I20260619 10:25:19.649787 6221 raft_consensus.cc:2454] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e35e0569a0546be9bcca81bb84a8350 in term 0.
I20260619 10:25:19.649825 5959 raft_consensus.cc:2454] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e35e0569a0546be9bcca81bb84a8350 in term 0.
I20260619 10:25:19.650035 6024 leader_election.cc:302] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f064faa82ea482da6108bb271f6a392, 4e35e0569a0546be9bcca81bb84a8350; no voters:
I20260619 10:25:19.650177 6289 raft_consensus.cc:2781] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:19.650249 6289 raft_consensus.cc:482] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:19.650265 6289 raft_consensus.cc:3037] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:19.651188 6289 raft_consensus.cc:504] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.651366 6289 leader_election.cc:288] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [CANDIDATE]: Term 1 election: Requested vote from peers d2838ef5e1aa44009d9384e5d04b6341 (127.19.65.131:41191), 0f064faa82ea482da6108bb271f6a392 (127.19.65.129:40669)
I20260619 10:25:19.651513 5959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5044709797a4cbb9ba31e82e1da073e" candidate_uuid: "4e35e0569a0546be9bcca81bb84a8350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f064faa82ea482da6108bb271f6a392"
I20260619 10:25:19.651558 6221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5044709797a4cbb9ba31e82e1da073e" candidate_uuid: "4e35e0569a0546be9bcca81bb84a8350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2838ef5e1aa44009d9384e5d04b6341"
I20260619 10:25:19.651592 5959 raft_consensus.cc:3037] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:19.651654 6221 raft_consensus.cc:3037] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:19.652346 5959 raft_consensus.cc:2454] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e35e0569a0546be9bcca81bb84a8350 in term 1.
I20260619 10:25:19.652503 6221 raft_consensus.cc:2454] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e35e0569a0546be9bcca81bb84a8350 in term 1.
I20260619 10:25:19.652590 6024 leader_election.cc:302] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f064faa82ea482da6108bb271f6a392, 4e35e0569a0546be9bcca81bb84a8350; no voters:
I20260619 10:25:19.652690 6289 raft_consensus.cc:2781] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:19.652871 6289 raft_consensus.cc:686] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 1 LEADER]: Becoming Leader. State: Replica: 4e35e0569a0546be9bcca81bb84a8350, State: Running, Role: LEADER
I20260619 10:25:19.652971 6289 consensus_queue.cc:237] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } }
I20260619 10:25:19.653803 5817 catalog_manager.cc:5495] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e35e0569a0546be9bcca81bb84a8350 (127.19.65.130). New cstate: current_term: 1 leader_uuid: "4e35e0569a0546be9bcca81bb84a8350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e35e0569a0546be9bcca81bb84a8350" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260619 10:25:19.712909 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.710751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.19.65.130:38587)
0619 10:25:19.710792 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:19.710794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:19.710919 (+ 125us) server_negotiation.cc:407] Connection header received
0619 10:25:19.710963 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:19.710964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:19.710982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:19.711039 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:19.711139 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.711614 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.712290 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:19.712474 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:19.712527 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:19.712643 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:19.712722 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:19.712788 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:19.712833 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 10:25:19.712840 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:25:19.714553 5959 raft_consensus.cc:1261] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 1 FOLLOWER]: Refusing update from remote peer 4e35e0569a0546be9bcca81bb84a8350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:19.714560 6221 raft_consensus.cc:1261] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 1 FOLLOWER]: Refusing update from remote peer 4e35e0569a0546be9bcca81bb84a8350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:19.714951 6289 consensus_queue.cc:1035] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2838ef5e1aa44009d9384e5d04b6341" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:19.715090 6291 consensus_queue.cc:1035] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f064faa82ea482da6108bb271f6a392" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:19.717242 6294 mvcc.cc:205] Tried to move back new op lower bound from 7298517891948359680 to 7298517891699122176. Current Snapshot: MvccSnapshot[applied={T|T < 7298517891948359680}]
I20260619 10:25:19.717583 6295 mvcc.cc:205] Tried to move back new op lower bound from 7298517891948359680 to 7298517891699122176. Current Snapshot: MvccSnapshot[applied={T|T < 7298517891948359680}]
I20260619 10:25:19.717753 6296 mvcc.cc:205] Tried to move back new op lower bound from 7298517891948359680 to 7298517891699122176. Current Snapshot: MvccSnapshot[applied={T|T < 7298517891948359680}]
I20260619 10:25:19.719275 5816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:19.719380 5816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:19.720099 5816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14109d25abd740018fb0142e0b1ee22b: Sending DeleteTablet for 3 replicas of tablet b5044709797a4cbb9ba31e82e1da073e
I20260619 10:25:19.720400 6070 tablet_service.cc:1514] Processing DeleteTablet for tablet b5044709797a4cbb9ba31e82e1da073e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:19 UTC) from {username='slave'} at 127.0.0.1:47590
I20260619 10:25:19.720440 6201 tablet_service.cc:1514] Processing DeleteTablet for tablet b5044709797a4cbb9ba31e82e1da073e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:19 UTC) from {username='slave'} at 127.0.0.1:51974
I20260619 10:25:19.720587 6304 tablet_replica.cc:330] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: stopping tablet replica
I20260619 10:25:19.720588 5939 tablet_service.cc:1514] Processing DeleteTablet for tablet b5044709797a4cbb9ba31e82e1da073e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:19 UTC) from {username='slave'} at 127.0.0.1:53838
I20260619 10:25:19.720669 6304 raft_consensus.cc:2229] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:19.720772 6304 raft_consensus.cc:2258] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:19.720783 6306 tablet_replica.cc:330] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392: stopping tablet replica
I20260619 10:25:19.720835 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5877
I20260619 10:25:19.720863 6306 raft_consensus.cc:2229] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:19.720969 6306 raft_consensus.cc:2258] T b5044709797a4cbb9ba31e82e1da073e P 0f064faa82ea482da6108bb271f6a392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:19.721163 6304 ts_tablet_manager.cc:1905] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:19.723459 6304 ts_tablet_manager.cc:1918] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:19.723520 6304 log.cc:1196] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/b5044709797a4cbb9ba31e82e1da073e
I20260619 10:25:19.723760 6304 ts_tablet_manager.cc:1939] T b5044709797a4cbb9ba31e82e1da073e P 4e35e0569a0546be9bcca81bb84a8350: Deleting consensus metadata
I20260619 10:25:19.724604 5801 catalog_manager.cc:4841] TS 4e35e0569a0546be9bcca81bb84a8350 (127.19.65.130:38587): tablet b5044709797a4cbb9ba31e82e1da073e (table test-table [id=8889c1225fbd43b496dfc9d26ada7c6c]) successfully deleted
I20260619 10:25:19.727394 6305 tablet_replica.cc:330] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: stopping tablet replica
I20260619 10:25:19.727491 6305 raft_consensus.cc:2229] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 10:25:19.727629 5802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:40669 (error 108)
I20260619 10:25:19.727672 6305 raft_consensus.cc:2258] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:19.728065 6305 ts_tablet_manager.cc:1905] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:19.728093 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:19.727985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:40669 (local address 127.0.0.1:45352)
0619 10:25:19.728028 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:19.728065 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:25:19.728111 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6008
W20260619 10:25:19.729373 5802 catalog_manager.cc:4584] TS 0f064faa82ea482da6108bb271f6a392 (127.19.65.129:40669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5044709797a4cbb9ba31e82e1da073e: Network error: Client connection negotiation failed: client connection to 127.19.65.129:40669: connect: Connection refused (error 111)
I20260619 10:25:19.730309 6305 ts_tablet_manager.cc:1918] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:19.730381 6305 log.cc:1196] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/b5044709797a4cbb9ba31e82e1da073e
I20260619 10:25:19.730600 6305 ts_tablet_manager.cc:1939] T b5044709797a4cbb9ba31e82e1da073e P d2838ef5e1aa44009d9384e5d04b6341: Deleting consensus metadata
I20260619 10:25:19.730970 5802 catalog_manager.cc:4841] TS d2838ef5e1aa44009d9384e5d04b6341 (127.19.65.131:41191): tablet b5044709797a4cbb9ba31e82e1da073e (table test-table [id=8889c1225fbd43b496dfc9d26ada7c6c]) successfully deleted
I20260619 10:25:19.732896 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6139
I20260619 10:25:19.737145 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 5786
2026-06-19T10:25:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (956 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:19 dist-test-slave-xd2q krb5kdc[6310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:19 dist-test-slave-xd2q krb5kdc[6310](info): set up 2 sockets
Jun 19 10:25:19 dist-test-slave-xd2q krb5kdc[6310](info): commencing operation
krb5kdc: starting...
W20260619 10:25:20.771010 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:20 dist-test-slave-xd2q krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:20.888211 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32853
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:20.967558 6323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:20.967757 6323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:20.967778 6323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:20.968837 6323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:20.968866 6323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:20.968880 6323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:20.968892 6323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:20.968904 6323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:20.970209 6323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:32853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:20.970381 6323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:20.970526 6323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:20.975265 6329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:20.975283 6331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:20.975380 6328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:20.975423 6323 server_base.cc:1034] running on GCE node
I20260619 10:25:20.975800 6323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:20.976073 6323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:20.977248 6323 hybrid_clock.cc:648] HybridClock initialized: now 1781864720977235 us; error 47 us; skew 500 ppm
Jun 19 10:25:20 dist-test-slave-xd2q krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864720, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:20.979784 6323 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:20.980257 6323 webserver.cc:460] Webserver started at http://127.19.65.190:40209/ using document root <none> and password file <none>
I20260619 10:25:20.980494 6323 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:20.980562 6323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:20.980691 6323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:20.981710 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "cdcf352feefa42f5a5e72a28ee121b86"
format_stamp: "Formatted at 2026-06-19 10:25:20 on dist-test-slave-xd2q"
server_key: "c4da415cc87cdcc56e00396b8046484b"
server_key_iv: "275e0ee0b5f4d598fdb46a6af9ffc60a"
server_key_version: "encryptionkey@0"
I20260619 10:25:20.982076 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "cdcf352feefa42f5a5e72a28ee121b86"
format_stamp: "Formatted at 2026-06-19 10:25:20 on dist-test-slave-xd2q"
server_key: "c4da415cc87cdcc56e00396b8046484b"
server_key_iv: "275e0ee0b5f4d598fdb46a6af9ffc60a"
server_key_version: "encryptionkey@0"
I20260619 10:25:20.983436 6323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 10:25:20.984134 6338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:20.984313 6323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:20.984416 6323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "cdcf352feefa42f5a5e72a28ee121b86"
format_stamp: "Formatted at 2026-06-19 10:25:20 on dist-test-slave-xd2q"
server_key: "c4da415cc87cdcc56e00396b8046484b"
server_key_iv: "275e0ee0b5f4d598fdb46a6af9ffc60a"
server_key_version: "encryptionkey@0"
I20260619 10:25:20.984499 6323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.000806 6323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.002516 6323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.002676 6323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.006644 6323 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:32853
I20260619 10:25:21.006677 6390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:32853 every 8 connection(s)
I20260619 10:25:21.007010 6323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:21.007849 6391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:21.010109 6391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: Bootstrap starting.
I20260619 10:25:21.010677 6391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:21.010967 6391 log.cc:824] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:21.011696 6391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: No bootstrap required, opened a new log
I20260619 10:25:21.012490 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6323
I20260619 10:25:21.012584 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:21.012801 19718 external_mini_cluster.cc:1428] Setting key eef06b76e256f6ef442a1341aa6c6261
I20260619 10:25:21.013880 6391 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } }
I20260619 10:25:21.014050 6391 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:21.014088 6391 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcf352feefa42f5a5e72a28ee121b86, State: Initialized, Role: FOLLOWER
I20260619 10:25:21.014238 6391 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } }
I20260619 10:25:21.014318 6391 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:21.014341 6391 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:21.014407 6391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:21.015535 6391 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } }
I20260619 10:25:21.015645 6391 leader_election.cc:302] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdcf352feefa42f5a5e72a28ee121b86; no voters:
I20260619 10:25:21.015805 6391 leader_election.cc:288] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:21.015862 6396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:21.016021 6391 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:21.016036 6396 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [term 1 LEADER]: Becoming Leader. State: Replica: cdcf352feefa42f5a5e72a28ee121b86, State: Running, Role: LEADER
I20260619 10:25:21.016125 6396 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } }
I20260619 10:25:21.016498 6396 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcf352feefa42f5a5e72a28ee121b86. Latest consensus state: current_term: 1 leader_uuid: "cdcf352feefa42f5a5e72a28ee121b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } } }
I20260619 10:25:21.016572 6396 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:21.016793 6405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:21.017225 6405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:21.017565 6398 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdcf352feefa42f5a5e72a28ee121b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcf352feefa42f5a5e72a28ee121b86" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32853 } } }
I20260619 10:25:21.017719 6398 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:21.019068 6405 catalog_manager.cc:1347] Generated new cluster ID: cb806a611bea462fb0620614676b6f23
I20260619 10:25:21.019136 6405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:21.022452 6405 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:21.022908 6405 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:21.024808 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.013340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.19.65.190:32853)
0619 10:25:21.013542 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.013552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.013589 (+ 37us) server_negotiation.cc:407] Connection header received
0619 10:25:21.013836 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.013844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.013906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.014059 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.014793 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.016280 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.017369 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.018019 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.018129 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.020693 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.020697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.020698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:21.020709 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.022691 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.023154 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.023157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.023157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.023278 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.023525 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.023526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.023526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.023795 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.023957 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.024459 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.024603 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260619 10:25:21.032943 6405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: Generated new TSK 0
I20260619 10:25:21.033107 6405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.65.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
Entry for principal kudu/127.19.65.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab.
I20260619 10:25:21.056094 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:21.139871 6417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:21.140069 6417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:21.140093 6417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:21.141250 6417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:21.141295 6417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:21.141324 6417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:21.142807 6417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.129.keytab
--principal=kudu/127.19.65.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:21.143051 6417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:21.143364 6417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:21.148274 6425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.148284 6422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.148295 6423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:21.148663 6417 server_base.cc:1034] running on GCE node
I20260619 10:25:21.148918 6417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:21.149161 6417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:21.150326 6417 hybrid_clock.cc:648] HybridClock initialized: now 1781864721150306 us; error 30 us; skew 500 ppm
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:21.152897 6417 init.cc:377] Logged in from keytab as kudu/127.19.65.129@KRBTEST.COM (short username kudu)
I20260619 10:25:21.153326 6417 webserver.cc:460] Webserver started at http://127.19.65.129:38527/ using document root <none> and password file <none>
I20260619 10:25:21.153564 6417 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:21.153631 6417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:21.153770 6417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:21.154812 6417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "6d076c6a385c44c6a3692b72fd4aa380"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "50d8de6507e068729d2eedd7573d7dba"
server_key_iv: "5381f64e1bde438225a9cfa5b381dda2"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.155165 6417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "6d076c6a385c44c6a3692b72fd4aa380"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "50d8de6507e068729d2eedd7573d7dba"
server_key_iv: "5381f64e1bde438225a9cfa5b381dda2"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.156504 6417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:21.157222 6432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.157399 6417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.157490 6417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "6d076c6a385c44c6a3692b72fd4aa380"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "50d8de6507e068729d2eedd7573d7dba"
server_key_iv: "5381f64e1bde438225a9cfa5b381dda2"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.157562 6417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.171809 6417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.174021 6417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.174176 6417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.174400 6417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:21.174696 6417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:21.174750 6417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.174789 6417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:21.174840 6417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.180896 6417 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:45425
I20260619 10:25:21.180896 6545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:45425 every 8 connection(s)
I20260619 10:25:21.181268 6417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.129@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:21.190200 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.181954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:57771 (local address 127.19.65.190:32853)
0619 10:25:21.182002 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.182004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.182178 (+ 174us) server_negotiation.cc:407] Connection header received
0619 10:25:21.182682 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.182684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.182696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.182772 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.183754 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.184312 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.185315 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.185532 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.185594 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.187707 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.187708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.187708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:21.187717 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.189093 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.189450 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.189451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.189451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.189501 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.189702 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.189702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.189702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.189799 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.189848 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.190061 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.190127 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:25:21.190336 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.181894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32853 (local address 127.19.65.129:57771)
0619 10:25:21.182046 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.182087 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.182548 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.182830 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.182833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.183446 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:21.183632 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.183636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.184398 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.184398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.185212 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.185214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.185610 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.185611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.185707 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.186242 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:21.186250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:21.187612 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:21.189168 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.189174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.189175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.189386 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.189557 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.189557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.189558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.189649 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.189852 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:21.189856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:21.189957 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.190035 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.190143 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260619 10:25:21.190836 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6417
I20260619 10:25:21.190897 6546 heartbeater.cc:347] Connected to a master server at 127.19.65.190:32853
I20260619 10:25:21.190958 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:21.190971 6546 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:21.191102 6546 heartbeater.cc:511] Master 127.19.65.190:32853 requested a full tablet report, sending...
I20260619 10:25:21.191108 19718 external_mini_cluster.cc:1428] Setting key 7af2f44f2dca4258b704c7fd7d175790
I20260619 10:25:21.191645 6355 ts_manager.cc:194] Registered new tserver with Master: 6d076c6a385c44c6a3692b72fd4aa380 (127.19.65.129:45425)
I20260619 10:25:21.192371 6355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.129@KRBTEST.COM'} at 127.19.65.129:57771
WARNING: no policy specified for kudu/127.19.65.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
Entry for principal kudu/127.19.65.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab.
I20260619 10:25:21.214769 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:21.294687 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:21.294924 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:21.294960 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:21.296125 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:21.296193 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:21.296236 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:21.297700 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.130.keytab
--principal=kudu/127.19.65.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:21.297951 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:21.298164 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:21.303067 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.303068 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.303118 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:21.303503 6551 server_base.cc:1034] running on GCE node
I20260619 10:25:21.303684 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:21.303910 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:21.305050 6551 hybrid_clock.cc:648] HybridClock initialized: now 1781864721305026 us; error 45 us; skew 500 ppm
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:21.307547 6551 init.cc:377] Logged in from keytab as kudu/127.19.65.130@KRBTEST.COM (short username kudu)
I20260619 10:25:21.308023 6551 webserver.cc:460] Webserver started at http://127.19.65.130:45251/ using document root <none> and password file <none>
I20260619 10:25:21.308244 6551 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:21.308311 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:21.308463 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:21.309464 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "9313fd500ce0477893f4d7bf0f2d651c"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "fdd57bf5ae93fbcd6e1deec849bec756"
server_key_iv: "81613a0af887179ce091392b945dfe5b"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.309816 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "9313fd500ce0477893f4d7bf0f2d651c"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "fdd57bf5ae93fbcd6e1deec849bec756"
server_key_iv: "81613a0af887179ce091392b945dfe5b"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.311033 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:21.311792 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.311986 6551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.312083 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "9313fd500ce0477893f4d7bf0f2d651c"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "fdd57bf5ae93fbcd6e1deec849bec756"
server_key_iv: "81613a0af887179ce091392b945dfe5b"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.312166 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.327842 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.330084 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.330245 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.330467 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:21.330775 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:21.330834 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.330874 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:21.330902 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.336838 6551 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:34295
I20260619 10:25:21.336884 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:34295 every 8 connection(s)
I20260619 10:25:21.337291 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:21.339958 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6551
I20260619 10:25:21.340071 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:21.340255 19718 external_mini_cluster.cc:1428] Setting key d7ff51df84b9d1e74437c4e26394ed7c
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.130@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:21.348191 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.338521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59443 (local address 127.19.65.190:32853)
0619 10:25:21.338566 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.338568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.338803 (+ 235us) server_negotiation.cc:407] Connection header received
0619 10:25:21.339240 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.339241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.339264 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.339395 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.340594 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.341138 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.342660 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.342988 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.343068 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.345183 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.345184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.345184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:21.345192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.346677 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.347164 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.347165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.347166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.347225 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.347566 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.347567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.347567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.347679 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.347738 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.348063 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.348125 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:25:21.348470 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.338474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32853 (local address 127.19.65.130:59443)
0619 10:25:21.338630 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.338686 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.339159 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.339449 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.339452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.340196 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:21.340489 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.340495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.341226 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.341227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.342559 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.342561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.343075 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.343076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.343168 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.343665 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:21.343673 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:21.345088 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:21.346761 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.346767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.346770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.347068 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.347332 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.347333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.347334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.347477 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.347753 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:21.347759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:21.347921 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.348045 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.348209 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260619 10:25:21.349009 6680 heartbeater.cc:347] Connected to a master server at 127.19.65.190:32853
I20260619 10:25:21.349104 6680 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.65.131@KRBTEST.COM; defaulting to no policy
I20260619 10:25:21.349310 6680 heartbeater.cc:511] Master 127.19.65.190:32853 requested a full tablet report, sending...
I20260619 10:25:21.349649 6355 ts_manager.cc:194] Registered new tserver with Master: 9313fd500ce0477893f4d7bf0f2d651c (127.19.65.130:34295)
I20260619 10:25:21.350093 6355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.130@KRBTEST.COM'} at 127.19.65.130:59443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
Entry for principal kudu/127.19.65.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab.
I20260619 10:25:21.365528 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:21.444446 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:21.444646 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:21.444669 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:21.445818 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:21.445851 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:21.445876 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:21.447180 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.131.keytab
--principal=kudu/127.19.65.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:21.447443 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:21.447624 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:21.452353 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.452363 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.452461 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:21.452505 6685 server_base.cc:1034] running on GCE node
I20260619 10:25:21.452807 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:21.453054 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:21.454267 6685 hybrid_clock.cc:648] HybridClock initialized: now 1781864721454230 us; error 37 us; skew 500 ppm
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:21.456982 6685 init.cc:377] Logged in from keytab as kudu/127.19.65.131@KRBTEST.COM (short username kudu)
I20260619 10:25:21.457466 6685 webserver.cc:460] Webserver started at http://127.19.65.131:43907/ using document root <none> and password file <none>
I20260619 10:25:21.457705 6685 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:21.457773 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:21.457929 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:21.458941 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "0efd1d017ff22b2606d4eae6d32e417b"
server_key_iv: "1a76b70c3aae250985d184875ec0327e"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.459336 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "0efd1d017ff22b2606d4eae6d32e417b"
server_key_iv: "1a76b70c3aae250985d184875ec0327e"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.460706 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:21.461580 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.461795 6685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:21.461898 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "0efd1d017ff22b2606d4eae6d32e417b"
server_key_iv: "1a76b70c3aae250985d184875ec0327e"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.461982 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.489144 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.491175 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.491361 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.491604 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:21.491945 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:21.491998 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.492033 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:21.492060 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.497671 6685 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:43701
I20260619 10:25:21.497713 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:43701 every 8 connection(s)
I20260619 10:25:21.498049 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:21.500283 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6685
I20260619 10:25:21.500393 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:21.500558 19718 external_mini_cluster.cc:1428] Setting key 24d7372b55d8010c2cfec0ccf9046b51
Jun 19 10:25:21 dist-test-slave-xd2q krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864721, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.131@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:21.507258 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.498754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:35991 (local address 127.19.65.190:32853)
0619 10:25:21.498857 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.498859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.499026 (+ 167us) server_negotiation.cc:407] Connection header received
0619 10:25:21.499597 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.499598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.499611 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.499664 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.500879 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.501443 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.502419 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.502620 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.502679 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.504739 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.504741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.504741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:21.504747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.506111 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.506471 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.506472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.506472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.506526 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.506752 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.506752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.506753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.506867 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.506924 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.507139 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.507196 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 10:25:21.507510 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.498673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:32853 (local address 127.19.65.131:35991)
0619 10:25:21.498889 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.498935 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.499496 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.499737 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.499740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.500464 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:21.500779 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.500785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.501557 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.501558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.502332 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.502334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.502697 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.502698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.502796 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.503196 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:21.503205 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:21.504659 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:21.506177 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.506181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.506183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.506372 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.506592 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:21.506593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:21.506593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:21.506681 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:21.506927 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:21.506931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:21.507035 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.507111 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.507221 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260619 10:25:21.507978 6814 heartbeater.cc:347] Connected to a master server at 127.19.65.190:32853
I20260619 10:25:21.508076 6814 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:21.508273 6814 heartbeater.cc:511] Master 127.19.65.190:32853 requested a full tablet report, sending...
I20260619 10:25:21.508682 6355 ts_manager.cc:194] Registered new tserver with Master: 9fb0a2f8d4eb4e418c2c10bd6c25651a (127.19.65.131:43701)
I20260619 10:25:21.509086 6355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.65.131@KRBTEST.COM'} at 127.19.65.131:35991
I20260619 10:25:21.512295 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:21.519696 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.513757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.19.65.190:32853)
0619 10:25:21.513816 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.513818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.513936 (+ 118us) server_negotiation.cc:407] Connection header received
0619 10:25:21.513975 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.513976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.513985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.514031 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.514543 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.515091 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.515961 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.516132 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.516185 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.517444 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.517445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.517445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:21.517451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.518762 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.519052 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.519052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.519053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.519098 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:21.519236 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:21.519236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:21.519236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:21.519410 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.519503 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.519586 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.519640 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:25:21.521073 6355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:21.522539 6355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:21.528537 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.524675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:43701 (local address 127.0.0.1:43416)
0619 10:25:21.524998 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.525009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.525060 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.525400 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.525400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.525402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:21.525442 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.525443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526233 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.526234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.527357 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.527359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528226 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.528324 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528336 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.528418 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.528469 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":40,"threads_started":1}
I20260619 10:25:21.528551 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.525085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.19.65.131:43701)
0619 10:25:21.525224 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.525229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.525237 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:21.525267 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.525269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.525297 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.525404 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:21.525497 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526139 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.527449 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528129 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528223 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528281 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.528444 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.528480 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260619 10:25:21.528962 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.524827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.19.65.130:34295)
0619 10:25:21.525012 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.525017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.525027 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:21.525128 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.525131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.525147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.525200 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:21.525457 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526607 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.527755 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528469 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528538 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528587 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.528801 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.528861 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260619 10:25:21.529150 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.524655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45425 (local address 127.0.0.1:54662)
0619 10:25:21.524758 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.524796 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.524858 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.525253 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.525257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.525259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:21.525325 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.525327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526734 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.526735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.527838 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.527840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528907 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.528974 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528978 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.529030 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.529077 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":23,"thread_start_us":49,"threads_started":1}
I20260619 10:25:21.529207 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.524655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:34295 (local address 127.0.0.1:55416)
0619 10:25:21.524985 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.524994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.525033 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.525307 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.525308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.525310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:21.525362 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.525363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526729 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.526729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.527677 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.527679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528624 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.528708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528712 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.529080 (+ 368us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.529112 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260619 10:25:21.529469 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.524924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.19.65.129:45425)
0619 10:25:21.525107 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.525111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.525120 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:21.525153 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.525155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.525171 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.525530 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:21.525600 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.526636 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.527916 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.528752 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.528809 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.528877 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.529320 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.529374 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260619 10:25:21.529371 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f (DEFAULT_TABLE table=test-table [id=93d47ff814454f87b92173c9a7794882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:21.529685 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8207b05c2ab345e3af09e78fef09ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:21.529844 6614 tablet_service.cc:1467] Processing CreateTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f (DEFAULT_TABLE table=test-table [id=93d47ff814454f87b92173c9a7794882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:21.530001 6480 tablet_service.cc:1467] Processing CreateTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f (DEFAULT_TABLE table=test-table [id=93d47ff814454f87b92173c9a7794882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:21.530143 6614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8207b05c2ab345e3af09e78fef09ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:21.530258 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8207b05c2ab345e3af09e78fef09ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:21.532784 6828 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: Bootstrap starting.
I20260619 10:25:21.533339 6830 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Bootstrap starting.
I20260619 10:25:21.533496 6828 tablet_bootstrap.cc:654] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:21.533797 6829 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Bootstrap starting.
I20260619 10:25:21.534017 6830 tablet_bootstrap.cc:654] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:21.534124 6828 log.cc:824] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:21.534420 6829 tablet_bootstrap.cc:654] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:21.534623 6830 log.cc:824] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:21.534967 6828 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: No bootstrap required, opened a new log
I20260619 10:25:21.534981 6829 log.cc:824] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:21.535053 6828 ts_tablet_manager.cc:1397] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:21.535423 6830 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: No bootstrap required, opened a new log
I20260619 10:25:21.535490 6830 ts_tablet_manager.cc:1397] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:21.535751 6829 tablet_bootstrap.cc:492] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: No bootstrap required, opened a new log
I20260619 10:25:21.535815 6829 ts_tablet_manager.cc:1397] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:21.537130 6829 raft_consensus.cc:348] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537081 6828 raft_consensus.cc:348] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537293 6828 raft_consensus.cc:374] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:21.537294 6829 raft_consensus.cc:374] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:21.537343 6829 raft_consensus.cc:729] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb0a2f8d4eb4e418c2c10bd6c25651a, State: Initialized, Role: FOLLOWER
I20260619 10:25:21.537343 6828 raft_consensus.cc:729] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d076c6a385c44c6a3692b72fd4aa380, State: Initialized, Role: FOLLOWER
I20260619 10:25:21.537451 6829 consensus_queue.cc:260] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537458 6828 consensus_queue.cc:260] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537355 6830 raft_consensus.cc:348] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537549 6830 raft_consensus.cc:374] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:21.537595 6830 raft_consensus.cc:729] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9313fd500ce0477893f4d7bf0f2d651c, State: Initialized, Role: FOLLOWER
I20260619 10:25:21.537720 6829 ts_tablet_manager.cc:1428] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:21.537730 6828 ts_tablet_manager.cc:1428] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:25:21.537720 6830 consensus_queue.cc:260] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.537868 6546 heartbeater.cc:503] Master 127.19.65.190:32853 was elected leader, sending a full tablet report...
I20260619 10:25:21.537885 6814 heartbeater.cc:503] Master 127.19.65.190:32853 was elected leader, sending a full tablet report...
I20260619 10:25:21.538232 6830 ts_tablet_manager.cc:1428] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:21.538341 6680 heartbeater.cc:503] Master 127.19.65.190:32853 was elected leader, sending a full tablet report...
W20260619 10:25:21.587863 6681 tablet.cc:2369] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:21.628278 6836 raft_consensus.cc:482] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:21.628384 6836 raft_consensus.cc:504] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.628793 6836 leader_election.cc:288] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d076c6a385c44c6a3692b72fd4aa380 (127.19.65.129:45425), 9fb0a2f8d4eb4e418c2c10bd6c25651a (127.19.65.131:43701)
I20260619 10:25:21.632720 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.629074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:48443 (local address 127.19.65.131:43701)
0619 10:25:21.629136 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.629138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.629290 (+ 152us) server_negotiation.cc:407] Connection header received
0619 10:25:21.629352 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.629354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.629363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.629439 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:21.629584 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.630360 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.631744 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.632341 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.632426 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.632435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.632608 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.632651 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:25:21.632720 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.629032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:34539 (local address 127.19.65.129:45425)
0619 10:25:21.629081 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.629083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.629093 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:21.629145 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.629146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.629160 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.629232 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:21.629415 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.630268 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.631744 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.632360 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.632428 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.632435 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.632608 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.632650 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:25:21.632740 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.628954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45425 (local address 127.19.65.130:34539)
0619 10:25:21.629006 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.629019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.629066 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.629248 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.629250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.629251 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:21.629331 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.629334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.630347 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.630347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.631629 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.631633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.632442 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.632443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.632525 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.632527 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.632585 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.632629 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":69}
I20260619 10:25:21.632748 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.629006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:43701 (local address 127.19.65.130:48443)
0619 10:25:21.629236 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.629246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.629290 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.629461 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.629462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.629464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:21.629509 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.629510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.630437 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.630437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.631630 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.631633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.632442 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.632443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.632522 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.632527 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.632585 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.632628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":9,"thread_start_us":151,"threads_started":1}
I20260619 10:25:21.633155 6500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8207b05c2ab345e3af09e78fef09ef5f" candidate_uuid: "9313fd500ce0477893f4d7bf0f2d651c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d076c6a385c44c6a3692b72fd4aa380" is_pre_election: true
I20260619 10:25:21.633340 6500 raft_consensus.cc:2454] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9313fd500ce0477893f4d7bf0f2d651c in term 0.
I20260619 10:25:21.633349 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8207b05c2ab345e3af09e78fef09ef5f" candidate_uuid: "9313fd500ce0477893f4d7bf0f2d651c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" is_pre_election: true
I20260619 10:25:21.633513 6768 raft_consensus.cc:2454] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9313fd500ce0477893f4d7bf0f2d651c in term 0.
I20260619 10:25:21.633597 6569 leader_election.cc:302] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d076c6a385c44c6a3692b72fd4aa380, 9313fd500ce0477893f4d7bf0f2d651c; no voters:
I20260619 10:25:21.633714 6836 raft_consensus.cc:2781] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:21.633766 6836 raft_consensus.cc:482] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:21.633797 6836 raft_consensus.cc:3037] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:21.634464 6836 raft_consensus.cc:504] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.634610 6836 leader_election.cc:288] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [CANDIDATE]: Term 1 election: Requested vote from peers 6d076c6a385c44c6a3692b72fd4aa380 (127.19.65.129:45425), 9fb0a2f8d4eb4e418c2c10bd6c25651a (127.19.65.131:43701)
I20260619 10:25:21.634727 6500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8207b05c2ab345e3af09e78fef09ef5f" candidate_uuid: "9313fd500ce0477893f4d7bf0f2d651c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d076c6a385c44c6a3692b72fd4aa380"
I20260619 10:25:21.634776 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8207b05c2ab345e3af09e78fef09ef5f" candidate_uuid: "9313fd500ce0477893f4d7bf0f2d651c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a"
I20260619 10:25:21.634933 6768 raft_consensus.cc:3037] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:21.634951 6500 raft_consensus.cc:3037] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:21.635775 6768 raft_consensus.cc:2454] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9313fd500ce0477893f4d7bf0f2d651c in term 1.
I20260619 10:25:21.635771 6500 raft_consensus.cc:2454] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9313fd500ce0477893f4d7bf0f2d651c in term 1.
I20260619 10:25:21.635973 6570 leader_election.cc:302] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9313fd500ce0477893f4d7bf0f2d651c, 9fb0a2f8d4eb4e418c2c10bd6c25651a; no voters:
I20260619 10:25:21.636078 6836 raft_consensus.cc:2781] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:21.636234 6836 raft_consensus.cc:686] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 1 LEADER]: Becoming Leader. State: Replica: 9313fd500ce0477893f4d7bf0f2d651c, State: Running, Role: LEADER
I20260619 10:25:21.636343 6836 consensus_queue.cc:237] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } }
I20260619 10:25:21.637178 6355 catalog_manager.cc:5495] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9313fd500ce0477893f4d7bf0f2d651c (127.19.65.130). New cstate: current_term: 1 leader_uuid: "9313fd500ce0477893f4d7bf0f2d651c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9313fd500ce0477893f4d7bf0f2d651c" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 34295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260619 10:25:21.642625 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.640252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.19.65.130:34295)
0619 10:25:21.640291 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.640293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.640302 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:21.640373 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.640374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.640392 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.640472 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:21.640629 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.641100 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.641862 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.642055 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.642131 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.642304 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:21.642393 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:21.642487 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.642507 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.642525 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:21.644243 6500 raft_consensus.cc:1261] T 8207b05c2ab345e3af09e78fef09ef5f P 6d076c6a385c44c6a3692b72fd4aa380 [term 1 FOLLOWER]: Refusing update from remote peer 9313fd500ce0477893f4d7bf0f2d651c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:21.644243 6768 raft_consensus.cc:1261] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 1 FOLLOWER]: Refusing update from remote peer 9313fd500ce0477893f4d7bf0f2d651c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:21.644595 6836 consensus_queue.cc:1035] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d076c6a385c44c6a3692b72fd4aa380" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:21.644735 6838 consensus_queue.cc:1035] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb0a2f8d4eb4e418c2c10bd6c25651a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:21.646931 6841 mvcc.cc:205] Tried to move back new op lower bound from 7298517899852480512 to 7298517899822989312. Current Snapshot: MvccSnapshot[applied={T|T < 7298517899852480512}]
I20260619 10:25:21.647594 6842 mvcc.cc:205] Tried to move back new op lower bound from 7298517899852480512 to 7298517899822989312. Current Snapshot: MvccSnapshot[applied={T|T < 7298517899852480512}]
I20260619 10:25:21.647782 6843 mvcc.cc:205] Tried to move back new op lower bound from 7298517899852480512 to 7298517899822989312. Current Snapshot: MvccSnapshot[applied={T|T < 7298517899852480512}]
I20260619 10:25:21.649115 6355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:21.649204 6355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:21.649915 6355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdcf352feefa42f5a5e72a28ee121b86: Sending DeleteTablet for 3 replicas of tablet 8207b05c2ab345e3af09e78fef09ef5f
I20260619 10:25:21.650205 6614 tablet_service.cc:1514] Processing DeleteTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:21 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:55416
I20260619 10:25:21.650247 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:21 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:43416
I20260619 10:25:21.650259 6480 tablet_service.cc:1514] Processing DeleteTablet for tablet 8207b05c2ab345e3af09e78fef09ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:21 UTC) from {username='kudu', principal='kudu/127.19.65.190@KRBTEST.COM'} at 127.0.0.1:54662
I20260619 10:25:21.650424 6852 tablet_replica.cc:330] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: stopping tablet replica
I20260619 10:25:21.650511 6852 raft_consensus.cc:2229] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:21.650599 6852 raft_consensus.cc:2258] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:21.650697 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6417
I20260619 10:25:21.651278 6851 tablet_replica.cc:330] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: stopping tablet replica
I20260619 10:25:21.651401 6851 raft_consensus.cc:2229] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:21.651501 6851 raft_consensus.cc:2258] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:21.651782 6852 ts_tablet_manager.cc:1905] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:21.653239 6851 ts_tablet_manager.cc:1905] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:21.653254 6852 ts_tablet_manager.cc:1918] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:21.653317 6852 log.cc:1196] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/8207b05c2ab345e3af09e78fef09ef5f
I20260619 10:25:21.653609 6852 ts_tablet_manager.cc:1939] T 8207b05c2ab345e3af09e78fef09ef5f P 9fb0a2f8d4eb4e418c2c10bd6c25651a: Deleting consensus metadata
I20260619 10:25:21.654006 6342 catalog_manager.cc:4841] TS 9fb0a2f8d4eb4e418c2c10bd6c25651a (127.19.65.131:43701): tablet 8207b05c2ab345e3af09e78fef09ef5f (table test-table [id=93d47ff814454f87b92173c9a7794882]) successfully deleted
I20260619 10:25:21.654701 6851 ts_tablet_manager.cc:1918] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:21.654758 6851 log.cc:1196] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/8207b05c2ab345e3af09e78fef09ef5f
I20260619 10:25:21.654946 6851 ts_tablet_manager.cc:1939] T 8207b05c2ab345e3af09e78fef09ef5f P 9313fd500ce0477893f4d7bf0f2d651c: Deleting consensus metadata
I20260619 10:25:21.655267 6340 catalog_manager.cc:4841] TS 9313fd500ce0477893f4d7bf0f2d651c (127.19.65.130:34295): tablet 8207b05c2ab345e3af09e78fef09ef5f (table test-table [id=93d47ff814454f87b92173c9a7794882]) successfully deleted
W20260619 10:25:21.657665 6341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:45425 (error 108)
I20260619 10:25:21.657888 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6551
I20260619 10:25:21.658499 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.657982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:45425 (local address 127.0.0.1:54670)
0619 10:25:21.658409 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.658457 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:45425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260619 10:25:21.659948 6341 catalog_manager.cc:4584] TS 6d076c6a385c44c6a3692b72fd4aa380 (127.19.65.129:45425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8207b05c2ab345e3af09e78fef09ef5f: Network error: Client connection negotiation failed: client connection to 127.19.65.129:45425: connect: Connection refused (error 111)
I20260619 10:25:21.663038 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6685
I20260619 10:25:21.667724 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6323
2026-06-19T10:25:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1931 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T10:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:21Z Disabled control of system clock
I20260619 10:25:21.694743 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43723
--webserver_interface=127.19.65.190
--webserver_port=0
--builtin_ntp_servers=127.19.65.148:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:43723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:21.773376 6858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:21.773608 6858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:21.773628 6858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:21.774698 6858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:21.774732 6858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:21.774745 6858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:21.774758 6858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:21.774770 6858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:21.776049 6858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:43723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:21.776232 6858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:21.776456 6858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:21.781195 6866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.781227 6863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.781219 6864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:21.781347 6858 server_base.cc:1034] running on GCE node
I20260619 10:25:21.781754 6858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:21.782048 6858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:21.783213 6858 hybrid_clock.cc:648] HybridClock initialized: now 1781864721783187 us; error 36 us; skew 500 ppm
I20260619 10:25:21.784813 6858 webserver.cc:460] Webserver started at http://127.19.65.190:46399/ using document root <none> and password file <none>
I20260619 10:25:21.785079 6858 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:21.785173 6858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:21.785342 6858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:21.786656 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "4e7c002559bc40978f1728542656a07a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "6c5d248482e481b254818c1fbc73b90e"
server_key_iv: "c49f0464586cbb3e593d47fae9653051"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.787063 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "4e7c002559bc40978f1728542656a07a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "6c5d248482e481b254818c1fbc73b90e"
server_key_iv: "c49f0464586cbb3e593d47fae9653051"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.788451 6858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:21.789260 6872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.789444 6858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:21.789542 6858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "4e7c002559bc40978f1728542656a07a"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "6c5d248482e481b254818c1fbc73b90e"
server_key_iv: "c49f0464586cbb3e593d47fae9653051"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.789628 6858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.816383 6858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.816757 6858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.816926 6858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.820792 6858 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:43723
I20260619 10:25:21.820830 6924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:43723 every 8 connection(s)
I20260619 10:25:21.821188 6858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:21.822113 6925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:21.824427 6925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: Bootstrap starting.
I20260619 10:25:21.824998 6925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:21.825277 6925 log.cc:824] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:21.826056 6925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: No bootstrap required, opened a new log
I20260619 10:25:21.827638 6925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } }
I20260619 10:25:21.827768 6925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:21.827790 6925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7c002559bc40978f1728542656a07a, State: Initialized, Role: FOLLOWER
I20260619 10:25:21.827963 6925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } }
I20260619 10:25:21.828022 6925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:21.828047 6925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:21.828112 6925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:21.828635 6925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } }
I20260619 10:25:21.828738 6925 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e7c002559bc40978f1728542656a07a; no voters:
I20260619 10:25:21.828949 6925 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:21.829017 6928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:21.829098 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6858
I20260619 10:25:21.829180 6928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [term 1 LEADER]: Becoming Leader. State: Replica: 4e7c002559bc40978f1728542656a07a, State: Running, Role: LEADER
I20260619 10:25:21.829200 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:21.829216 6925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:21.829299 6928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } }
I20260619 10:25:21.829406 19718 external_mini_cluster.cc:1428] Setting key 46770eaea8ceab987eaba63596599324
I20260619 10:25:21.829825 6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e7c002559bc40978f1728542656a07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } } }
I20260619 10:25:21.829905 6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:21.830134 6930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7c002559bc40978f1728542656a07a. Latest consensus state: current_term: 1 leader_uuid: "4e7c002559bc40978f1728542656a07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7c002559bc40978f1728542656a07a" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 43723 } } }
I20260619 10:25:21.830246 6930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:21.830399 6938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:21.831120 6938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:21.832674 6938 catalog_manager.cc:1347] Generated new cluster ID: db6b00bd110b4b68b87863060c550927
I20260619 10:25:21.832729 6938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:21.835059 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.830036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.19.65.190:43723)
0619 10:25:21.830218 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.830227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.830279 (+ 52us) server_negotiation.cc:407] Connection header received
0619 10:25:21.830401 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.830407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.830478 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.830971 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.831060 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.832237 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.833154 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.833598 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.833733 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.834418 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.834422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.834422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:21.834592 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.834624 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:21.834635 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.834713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.834795 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.834877 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260619 10:25:21.839594 6938 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:21.840243 6938 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:21.845335 6938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: Generated new TSK 0
I20260619 10:25:21.845481 6938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:21.850942 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.129:0
--local_ip_for_outbound_sockets=127.19.65.129
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--builtin_ntp_servers=127.19.65.148:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:21.943783 6949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:21.944010 6949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:21.944046 6949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:21.945158 6949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:21.945222 6949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:21.945272 6949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.129
I20260619 10:25:21.946668 6949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.65.129
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.129
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:21.946942 6949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:21.947223 6949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:21.952932 6955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.952951 6957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:21.952924 6954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:21.953312 6949 server_base.cc:1034] running on GCE node
I20260619 10:25:21.953521 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:21.953773 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:21.954958 6949 hybrid_clock.cc:648] HybridClock initialized: now 1781864721954945 us; error 26 us; skew 500 ppm
I20260619 10:25:21.956549 6949 webserver.cc:460] Webserver started at http://127.19.65.129:35087/ using document root <none> and password file <none>
I20260619 10:25:21.956792 6949 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:21.956859 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:21.956987 6949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:21.958360 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data/instance:
uuid: "e86eec955ef045299962adfdcbd45e46"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "5aa9b2526755dd53d964c02b8d7b2f70"
server_key_iv: "2f5c44fac744d93176b1c61bb74c7625"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.958761 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance:
uuid: "e86eec955ef045299962adfdcbd45e46"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "5aa9b2526755dd53d964c02b8d7b2f70"
server_key_iv: "2f5c44fac744d93176b1c61bb74c7625"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.960434 6949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 10:25:21.961337 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.961547 6949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:21.961642 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal
uuid: "e86eec955ef045299962adfdcbd45e46"
format_stamp: "Formatted at 2026-06-19 10:25:21 on dist-test-slave-xd2q"
server_key: "5aa9b2526755dd53d964c02b8d7b2f70"
server_key_iv: "2f5c44fac744d93176b1c61bb74c7625"
server_key_version: "encryptionkey@0"
I20260619 10:25:21.961712 6949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:21.978619 6949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:21.978952 6949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:21.979080 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:21.979379 6949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:21.979708 6949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:21.979745 6949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.979775 6949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:21.979794 6949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:21.986759 6949 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.129:46009
I20260619 10:25:21.986793 7076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.129:46009 every 8 connection(s)
I20260619 10:25:21.987149 6949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:21.992108 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.987881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.129:36163 (local address 127.19.65.190:43723)
0619 10:25:21.987944 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:21.987946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:21.988211 (+ 265us) server_negotiation.cc:407] Connection header received
0619 10:25:21.988668 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:21.988669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:21.988679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:21.988729 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:21.989402 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.989879 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.990851 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.991023 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.991071 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.991874 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:21.991875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:21.991876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:21.991879 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:21.991893 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:21.991898 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:21.991939 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:21.992028 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 10:25:21.992051 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:21.992403 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:21.987898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43723 (local address 127.19.65.129:36163)
0619 10:25:21.988087 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:21.988119 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:21.988590 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:21.988781 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:21.988785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:21.988792 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:21.989328 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.989332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.989955 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.989956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.990773 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:21.990775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:21.991092 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:21.991093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:21.991197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:21.991747 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:21.991747 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:21.991775 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:21.991797 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:21.991940 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:21.991944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:21.991946 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:25:21.992029 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:25:21.992111 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260619 10:25:21.992923 7077 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43723
I20260619 10:25:21.993000 7077 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:21.993201 7077 heartbeater.cc:511] Master 127.19.65.190:43723 requested a full tablet report, sending...
I20260619 10:25:21.993749 6889 ts_manager.cc:194] Registered new tserver with Master: e86eec955ef045299962adfdcbd45e46 (127.19.65.129:46009)
I20260619 10:25:21.994331 6889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.129:36163
I20260619 10:25:21.996573 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 6949
I20260619 10:25:21.996677 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:21.996835 19718 external_mini_cluster.cc:1428] Setting key 708398784d7ff779f34eea01a751055a
I20260619 10:25:21.997702 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.130:0
--local_ip_for_outbound_sockets=127.19.65.130
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--builtin_ntp_servers=127.19.65.148:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:22.077885 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:22.078070 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:22.078091 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:22.079174 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:22.079207 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:22.079231 7080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.130
I20260619 10:25:22.080495 7080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.65.130
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.130
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:22.080667 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:22.080894 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:22.085722 7088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:22.085726 7085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:22.085955 7080 server_base.cc:1034] running on GCE node
W20260619 10:25:22.085718 7086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:22.086215 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:22.086453 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:22.087617 7080 hybrid_clock.cc:648] HybridClock initialized: now 1781864722087597 us; error 36 us; skew 500 ppm
I20260619 10:25:22.088943 7080 webserver.cc:460] Webserver started at http://127.19.65.130:41913/ using document root <none> and password file <none>
I20260619 10:25:22.089123 7080 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:22.089164 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:22.089250 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:22.090195 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data/instance:
uuid: "a14fdbe622594b04aec679b8f4c7db54"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "520284355323427d60432f8e4c594fb8"
server_key_iv: "cff27e2226b20c1eb1fc1d8dc54b9345"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.090479 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance:
uuid: "a14fdbe622594b04aec679b8f4c7db54"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "520284355323427d60432f8e4c594fb8"
server_key_iv: "cff27e2226b20c1eb1fc1d8dc54b9345"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.091708 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:22.092370 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.092558 7080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:22.092643 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal
uuid: "a14fdbe622594b04aec679b8f4c7db54"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "520284355323427d60432f8e4c594fb8"
server_key_iv: "cff27e2226b20c1eb1fc1d8dc54b9345"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.092697 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:22.107448 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:22.107668 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:22.107762 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:22.107928 7080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:22.108175 7080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:22.108201 7080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.108220 7080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:22.108232 7080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.113471 7080 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.130:39721
I20260619 10:25:22.113570 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.130:39721 every 8 connection(s)
I20260619 10:25:22.113857 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:22.118993 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.114549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.130:59039 (local address 127.19.65.190:43723)
0619 10:25:22.114613 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.114615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.114858 (+ 243us) server_negotiation.cc:407] Connection header received
0619 10:25:22.115397 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.115399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.115409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.115466 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:22.116145 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.116656 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.117623 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.117853 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.117945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.118716 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:22.118718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:22.118718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:22.118723 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:22.118739 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:22.118743 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:22.118791 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.118883 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.118917 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:25:22.119217 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.114522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43723 (local address 127.19.65.130:59039)
0619 10:25:22.114743 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.114774 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.115251 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.115537 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.115541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.115550 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:22.116051 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.116055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.116752 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.116753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.117529 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.117531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.117958 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.117959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.118093 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.118582 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:22.118582 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:22.118609 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:22.118630 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:22.118791 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:22.118796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:22.118799 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.118886 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.118981 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260619 10:25:22.119731 7208 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43723
I20260619 10:25:22.119850 7208 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:22.120033 7208 heartbeater.cc:511] Master 127.19.65.190:43723 requested a full tablet report, sending...
I20260619 10:25:22.120397 6889 ts_manager.cc:194] Registered new tserver with Master: a14fdbe622594b04aec679b8f4c7db54 (127.19.65.130:39721)
I20260619 10:25:22.120723 6889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.130:59039
I20260619 10:25:22.122395 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7080
I20260619 10:25:22.122478 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:22.122624 19718 external_mini_cluster.cc:1428] Setting key 7828ae1f790968574a6905a466736592
I20260619 10:25:22.123515 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.65.131:0
--local_ip_for_outbound_sockets=127.19.65.131
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--builtin_ntp_servers=127.19.65.148:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:25:22.203231 7211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:22.203513 7211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:22.203536 7211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:22.204715 7211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:22.204753 7211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:22.204777 7211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.65.131
I20260619 10:25:22.206096 7211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.65.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.65.131
--webserver_port=0
--tserver_master_addrs=127.19.65.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.65.131
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:22.206341 7211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:22.206641 7211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:22.211854 7219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:22.211869 7216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:22.211848 7217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:22.212055 7211 server_base.cc:1034] running on GCE node
I20260619 10:25:22.212363 7211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:22.212621 7211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:22.213840 7211 hybrid_clock.cc:648] HybridClock initialized: now 1781864722213824 us; error 35 us; skew 500 ppm
I20260619 10:25:22.215085 7211 webserver.cc:460] Webserver started at http://127.19.65.131:38473/ using document root <none> and password file <none>
I20260619 10:25:22.215427 7211 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:22.215502 7211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:22.215637 7211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:22.216753 7211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data/instance:
uuid: "b917d384fc1f4a769b73b82007ed467a"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "23649f6ae1d260c9fab2f7b3b20180ed"
server_key_iv: "e7efc573da21c2b86bac0f173d0e633c"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.217123 7211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance:
uuid: "b917d384fc1f4a769b73b82007ed467a"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "23649f6ae1d260c9fab2f7b3b20180ed"
server_key_iv: "e7efc573da21c2b86bac0f173d0e633c"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.218391 7211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:22.219125 7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.219338 7211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:22.219455 7211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal
uuid: "b917d384fc1f4a769b73b82007ed467a"
format_stamp: "Formatted at 2026-06-19 10:25:22 on dist-test-slave-xd2q"
server_key: "23649f6ae1d260c9fab2f7b3b20180ed"
server_key_iv: "e7efc573da21c2b86bac0f173d0e633c"
server_key_version: "encryptionkey@0"
I20260619 10:25:22.219586 7211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:22.238039 7211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:22.238400 7211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:22.238556 7211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:22.238826 7211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:22.239149 7211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:22.239204 7211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.239246 7211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:22.239282 7211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:22.245122 7211 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.131:46587
I20260619 10:25:22.245162 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.131:46587 every 8 connection(s)
I20260619 10:25:22.245507 7211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:22.248119 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7211
I20260619 10:25:22.248236 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:22.248448 19718 external_mini_cluster.cc:1428] Setting key 094eb540cbf84ae3d098dd99982baac7
I20260619 10:25:22.251021 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.246229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:39651 (local address 127.19.65.190:43723)
0619 10:25:22.246272 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.246275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.246498 (+ 223us) server_negotiation.cc:407] Connection header received
0619 10:25:22.247037 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.247038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.247052 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.247112 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:22.247868 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.248488 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.249756 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.249947 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.250003 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.250718 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:22.250719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:22.250719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:22.250724 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:22.250737 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:22.250740 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:22.250784 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.250899 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.250963 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:22.251189 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.246197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:43723 (local address 127.19.65.131:39651)
0619 10:25:22.246378 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.246415 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.246930 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.247182 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.247185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.247195 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:22.247769 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.247774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.248590 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.248591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.249672 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.249675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.250024 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.250025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.250128 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.250586 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:25:22.250586 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:22.250614 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:25:22.250634 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:22.250799 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:22.250829 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:22.250832 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.250887 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.250996 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260619 10:25:22.251644 7339 heartbeater.cc:347] Connected to a master server at 127.19.65.190:43723
I20260619 10:25:22.251716 7339 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:22.251853 7339 heartbeater.cc:511] Master 127.19.65.190:43723 requested a full tablet report, sending...
I20260619 10:25:22.252140 6889 ts_manager.cc:194] Registered new tserver with Master: b917d384fc1f4a769b73b82007ed467a (127.19.65.131:46587)
I20260619 10:25:22.252507 6889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.65.131:39651
I20260619 10:25:22.259991 19718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:22.264021 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.261382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.19.65.190:43723)
0619 10:25:22.261442 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.261443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.261563 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:25:22.261623 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.261624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.261634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.261684 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:22.261805 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.262254 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.262927 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.263108 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.263155 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.263777 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:22.263778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:22.263778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:22.263782 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:22.263794 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:22.263797 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:22.263839 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.263874 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.263893 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:22.265558 6889 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:22.267125 6889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:22.273032 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.131:46587 (local address 127.0.0.1:42754)
0619 10:25:22.269276 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.269323 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.269383 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.269614 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.269617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.269619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:22.269684 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.269685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.270931 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.270932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.271972 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.271974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.272704 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.272785 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.272827 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.272896 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.272945 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260619 10:25:22.273154 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46009 (local address 127.0.0.1:52974)
0619 10:25:22.269684 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.269693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.269747 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.269956 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.269957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.269960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:22.269998 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.269999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.270899 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.270901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.272108 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.272111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.272900 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.272966 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.272969 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.273023 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.273061 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260619 10:25:22.273281 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.19.65.131:46587)
0619 10:25:22.269452 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.269455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.269464 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:22.269501 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.269503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.269514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.269607 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:22.269755 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.270829 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272044 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.272620 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272692 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.272776 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.273140 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.273184 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260619 10:25:22.273535 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39721 (local address 127.0.0.1:51934)
0619 10:25:22.269825 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.269834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.269872 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.270116 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.270117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.270118 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:22.270148 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.270150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.271311 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.271312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.272299 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.272302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.273296 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.273297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.273364 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.273368 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.273420 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.273457 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":64,"threads_started":1}
I20260619 10:25:22.273584 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.19.65.129:46009)
0619 10:25:22.269420 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.269425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.269745 (+ 320us) server_negotiation.cc:407] Connection header received
0619 10:25:22.269841 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.269845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.269860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.270067 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:22.270121 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.270815 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272199 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.272792 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272897 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.272977 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.273034 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.273506 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260619 10:25:22.273818 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.269500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.19.65.130:39721)
0619 10:25:22.269658 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.269662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.269927 (+ 265us) server_negotiation.cc:407] Connection header received
0619 10:25:22.269967 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.269969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.269984 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.270051 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:22.270232 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.271212 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.272377 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.273107 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.273168 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.273252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.273706 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.273747 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260619 10:25:22.273969 7273 tablet_service.cc:1467] Processing CreateTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 (DEFAULT_TABLE table=test-table [id=95c8bb960d694cdd8409654210cdbdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:22.274108 7011 tablet_service.cc:1467] Processing CreateTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 (DEFAULT_TABLE table=test-table [id=95c8bb960d694cdd8409654210cdbdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:22.274313 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e75575f79534d549ca4fc00ce1b1831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:22.274399 7011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e75575f79534d549ca4fc00ce1b1831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:22.277580 7353 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: Bootstrap starting.
I20260619 10:25:22.278160 7142 tablet_service.cc:1467] Processing CreateTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 (DEFAULT_TABLE table=test-table [id=95c8bb960d694cdd8409654210cdbdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:22.278252 7353 tablet_bootstrap.cc:654] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:22.278463 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e75575f79534d549ca4fc00ce1b1831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:22.278597 7353 log.cc:824] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:22.278911 7354 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Bootstrap starting.
I20260619 10:25:22.279418 7353 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: No bootstrap required, opened a new log
I20260619 10:25:22.279561 7354 tablet_bootstrap.cc:654] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:22.279556 7353 ts_tablet_manager.cc:1397] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:22.279867 7354 log.cc:824] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:22.280682 7354 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: No bootstrap required, opened a new log
I20260619 10:25:22.280750 7354 ts_tablet_manager.cc:1397] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:22.281337 7357 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Bootstrap starting.
I20260619 10:25:22.281894 7357 tablet_bootstrap.cc:654] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:22.282210 7357 log.cc:824] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:22.282300 7353 raft_consensus.cc:348] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.282467 7353 raft_consensus.cc:374] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:22.282514 7353 raft_consensus.cc:729] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86eec955ef045299962adfdcbd45e46, State: Initialized, Role: FOLLOWER
I20260619 10:25:22.282621 7353 consensus_queue.cc:260] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.282868 7353 ts_tablet_manager.cc:1428] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:22.282927 7357 tablet_bootstrap.cc:492] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: No bootstrap required, opened a new log
I20260619 10:25:22.282979 7357 ts_tablet_manager.cc:1397] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:22.282996 7077 heartbeater.cc:503] Master 127.19.65.190:43723 was elected leader, sending a full tablet report...
I20260619 10:25:22.283550 7354 raft_consensus.cc:348] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.283686 7354 raft_consensus.cc:374] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:22.283730 7354 raft_consensus.cc:729] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b917d384fc1f4a769b73b82007ed467a, State: Initialized, Role: FOLLOWER
I20260619 10:25:22.283924 7354 consensus_queue.cc:260] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.284168 7354 ts_tablet_manager.cc:1428] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:22.284230 7339 heartbeater.cc:503] Master 127.19.65.190:43723 was elected leader, sending a full tablet report...
I20260619 10:25:22.285188 7357 raft_consensus.cc:348] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.285333 7357 raft_consensus.cc:374] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:22.285378 7357 raft_consensus.cc:729] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14fdbe622594b04aec679b8f4c7db54, State: Initialized, Role: FOLLOWER
I20260619 10:25:22.285481 7357 consensus_queue.cc:260] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.285722 7357 ts_tablet_manager.cc:1428] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:25:22.285815 7208 heartbeater.cc:503] Master 127.19.65.190:43723 was elected leader, sending a full tablet report...
I20260619 10:25:22.301206 7360 raft_consensus.cc:482] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:22.301326 7360 raft_consensus.cc:504] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.301707 7360 leader_election.cc:288] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e86eec955ef045299962adfdcbd45e46 (127.19.65.129:46009), a14fdbe622594b04aec679b8f4c7db54 (127.19.65.130:39721)
I20260619 10:25:22.305171 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.301892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46009 (local address 127.19.65.131:43917)
0619 10:25:22.302068 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.302082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.302159 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.302316 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.302316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.302318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:22.302390 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.302391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.303033 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.303034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.304173 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.304175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.304882 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.304957 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.304960 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.305049 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.305093 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3}
I20260619 10:25:22.305159 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.302078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:43917 (local address 127.19.65.129:46009)
0619 10:25:22.302124 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.302126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.302154 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:25:22.302216 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.302217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.302230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.302309 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:22.302461 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.302947 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304241 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.304795 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304875 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.304884 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.305056 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.305093 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:25:22.305370 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.302086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.65.131:40781 (local address 127.19.65.130:39721)
0619 10:25:22.302133 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.302153 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.302275 (+ 122us) server_negotiation.cc:407] Connection header received
0619 10:25:22.302340 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.302341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.302355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.302425 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:22.302575 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.303025 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304461 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.304864 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304964 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.304972 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.305151 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.305190 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:25:22.305505 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.302023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.130:39721 (local address 127.19.65.131:40781)
0619 10:25:22.302203 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.302218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:22.302288 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:22.302432 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:22.302433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:22.302434 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:22.302500 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.302501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.303106 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.303107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.304332 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:22.304335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.304968 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.304969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:22.305061 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.305066 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:22.305142 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 10:25:22.305178 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260619 10:25:22.305495 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e75575f79534d549ca4fc00ce1b1831" candidate_uuid: "b917d384fc1f4a769b73b82007ed467a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86eec955ef045299962adfdcbd45e46" is_pre_election: true
I20260619 10:25:22.305656 7031 raft_consensus.cc:2454] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b917d384fc1f4a769b73b82007ed467a in term 0.
I20260619 10:25:22.305892 7162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e75575f79534d549ca4fc00ce1b1831" candidate_uuid: "b917d384fc1f4a769b73b82007ed467a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fdbe622594b04aec679b8f4c7db54" is_pre_election: true
I20260619 10:25:22.305928 7229 leader_election.cc:302] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b917d384fc1f4a769b73b82007ed467a, e86eec955ef045299962adfdcbd45e46; no voters:
I20260619 10:25:22.306011 7162 raft_consensus.cc:2454] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b917d384fc1f4a769b73b82007ed467a in term 0.
I20260619 10:25:22.306063 7360 raft_consensus.cc:2781] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:22.306113 7360 raft_consensus.cc:482] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:22.306142 7360 raft_consensus.cc:3037] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:22.306842 7360 raft_consensus.cc:504] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.306986 7360 leader_election.cc:288] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [CANDIDATE]: Term 1 election: Requested vote from peers e86eec955ef045299962adfdcbd45e46 (127.19.65.129:46009), a14fdbe622594b04aec679b8f4c7db54 (127.19.65.130:39721)
I20260619 10:25:22.307188 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e75575f79534d549ca4fc00ce1b1831" candidate_uuid: "b917d384fc1f4a769b73b82007ed467a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86eec955ef045299962adfdcbd45e46"
I20260619 10:25:22.307258 7162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e75575f79534d549ca4fc00ce1b1831" candidate_uuid: "b917d384fc1f4a769b73b82007ed467a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fdbe622594b04aec679b8f4c7db54"
I20260619 10:25:22.307284 7031 raft_consensus.cc:3037] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:22.307379 7162 raft_consensus.cc:3037] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:22.308020 7031 raft_consensus.cc:2454] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b917d384fc1f4a769b73b82007ed467a in term 1.
I20260619 10:25:22.308269 7229 leader_election.cc:302] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b917d384fc1f4a769b73b82007ed467a, e86eec955ef045299962adfdcbd45e46; no voters:
I20260619 10:25:22.308388 7360 raft_consensus.cc:2781] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:22.308490 7162 raft_consensus.cc:2454] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b917d384fc1f4a769b73b82007ed467a in term 1.
I20260619 10:25:22.308568 7360 raft_consensus.cc:686] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 1 LEADER]: Becoming Leader. State: Replica: b917d384fc1f4a769b73b82007ed467a, State: Running, Role: LEADER
I20260619 10:25:22.308679 7360 consensus_queue.cc:237] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } }
I20260619 10:25:22.309513 6889 catalog_manager.cc:5495] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a reported cstate change: term changed from 0 to 1, leader changed from <none> to b917d384fc1f4a769b73b82007ed467a (127.19.65.131). New cstate: current_term: 1 leader_uuid: "b917d384fc1f4a769b73b82007ed467a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b917d384fc1f4a769b73b82007ed467a" member_type: VOTER last_known_addr { host: "127.19.65.131" port: 46587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 } health_report { overall_health: UNKNOWN } } }
I20260619 10:25:22.317955 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.315495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.19.65.131:46587)
0619 10:25:22.315536 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:22.315538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:22.315568 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:25:22.315652 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:22.315653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:22.315669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:22.315736 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:22.315851 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.316304 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.317054 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:22.317262 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:22.317346 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:22.317531 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:22.317652 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:22.317758 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:22.317828 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:22.317848 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:22.319675 7031 raft_consensus.cc:1261] T 3e75575f79534d549ca4fc00ce1b1831 P e86eec955ef045299962adfdcbd45e46 [term 1 FOLLOWER]: Refusing update from remote peer b917d384fc1f4a769b73b82007ed467a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:22.319675 7162 raft_consensus.cc:1261] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 1 FOLLOWER]: Refusing update from remote peer b917d384fc1f4a769b73b82007ed467a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:22.320029 7363 consensus_queue.cc:1035] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e86eec955ef045299962adfdcbd45e46" member_type: VOTER last_known_addr { host: "127.19.65.129" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:22.320158 7360 consensus_queue.cc:1035] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14fdbe622594b04aec679b8f4c7db54" member_type: VOTER last_known_addr { host: "127.19.65.130" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:22.323341 7367 mvcc.cc:205] Tried to move back new op lower bound from 7298517902618906624 to 7298517902577082368. Current Snapshot: MvccSnapshot[applied={T|T < 7298517902618906624}]
I20260619 10:25:22.324590 6887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:22.324697 6887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:22.325441 6887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e7c002559bc40978f1728542656a07a: Sending DeleteTablet for 3 replicas of tablet 3e75575f79534d549ca4fc00ce1b1831
I20260619 10:25:22.325750 7011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:22 UTC) from {username='slave'} at 127.0.0.1:52974
I20260619 10:25:22.325769 7273 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:22 UTC) from {username='slave'} at 127.0.0.1:42754
I20260619 10:25:22.326020 7376 tablet_replica.cc:330] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: stopping tablet replica
I20260619 10:25:22.326035 7142 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e75575f79534d549ca4fc00ce1b1831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:22 UTC) from {username='slave'} at 127.0.0.1:51934
I20260619 10:25:22.326122 7376 raft_consensus.cc:2229] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:22.326225 7376 raft_consensus.cc:2258] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:22.326464 7377 tablet_replica.cc:330] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: stopping tablet replica
I20260619 10:25:22.326561 7377 raft_consensus.cc:2229] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:22.326689 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6949
I20260619 10:25:22.326658 7376 ts_tablet_manager.cc:1905] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:22.329488 7376 ts_tablet_manager.cc:1918] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:22.329566 7376 log.cc:1196] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-2/wal/wals/3e75575f79534d549ca4fc00ce1b1831
I20260619 10:25:22.329882 7376 ts_tablet_manager.cc:1939] T 3e75575f79534d549ca4fc00ce1b1831 P b917d384fc1f4a769b73b82007ed467a: Deleting consensus metadata
I20260619 10:25:22.330282 7377 raft_consensus.cc:2258] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:22.330394 6874 catalog_manager.cc:4841] TS b917d384fc1f4a769b73b82007ed467a (127.19.65.131:46587): tablet 3e75575f79534d549ca4fc00ce1b1831 (table test-table [id=95c8bb960d694cdd8409654210cdbdab]) successfully deleted
I20260619 10:25:22.330794 7377 ts_tablet_manager.cc:1905] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:22.332121 7377 ts_tablet_manager.cc:1918] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:22.332176 7377 log.cc:1196] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Deleting WAL directory at /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864489541676-19718-0/minicluster-data/ts-1/wal/wals/3e75575f79534d549ca4fc00ce1b1831
I20260619 10:25:22.332441 7377 ts_tablet_manager.cc:1939] T 3e75575f79534d549ca4fc00ce1b1831 P a14fdbe622594b04aec679b8f4c7db54: Deleting consensus metadata
I20260619 10:25:22.332803 6876 catalog_manager.cc:4841] TS a14fdbe622594b04aec679b8f4c7db54 (127.19.65.130:39721): tablet 3e75575f79534d549ca4fc00ce1b1831 (table test-table [id=95c8bb960d694cdd8409654210cdbdab]) successfully deleted
W20260619 10:25:22.337996 6876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.65.129:46009 (error 108)
I20260619 10:25:22.338244 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7080
I20260619 10:25:22.339435 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:22.338326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.129:46009 (local address 127.0.0.1:52988)
0619 10:25:22.339355 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:22.339400 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1009}
W20260619 10:25:22.339547 6876 catalog_manager.cc:4584] TS e86eec955ef045299962adfdcbd45e46 (127.19.65.129:46009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e75575f79534d549ca4fc00ce1b1831: Network error: Client connection negotiation failed: client connection to 127.19.65.129:46009: connect: Connection refused (error 111)
I20260619 10:25:22.342792 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7211
I20260619 10:25:22.346951 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 6858
2026-06-19T10:25:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (678 ms)
[----------] 6 tests from EncryptionPolicyTest (8032 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:22 dist-test-slave-xd2q krb5kdc[7381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:22 dist-test-slave-xd2q krb5kdc[7381](info): set up 2 sockets
Jun 19 10:25:22 dist-test-slave-xd2q krb5kdc[7381](info): commencing operation
krb5kdc: starting...
W20260619 10:25:23.383342 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:23.500514 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:23.579618 7394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:23.579854 7394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:23.579891 7394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:23.581142 7394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:23.581211 7394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:23.581238 7394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:23.581267 7394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:23.581297 7394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:23.582772 7394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:23.583053 7394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:23.583273 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:23.588315 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:23.588472 7394 server_base.cc:1034] running on GCE node
W20260619 10:25:23.588385 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:23.588325 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:23.588820 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:23.589140 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:23.590308 7394 hybrid_clock.cc:648] HybridClock initialized: now 1781864723590294 us; error 28 us; skew 500 ppm
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:23.592888 7394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:23.593253 7394 webserver.cc:460] Webserver started at http://127.0.0.1:43707/ using document root <none> and password file <none>
I20260619 10:25:23.593441 7394 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:23.593510 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:23.593596 7394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:23.594576 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "48f521422f554d03b7b595f3b1a62001"
format_stamp: "Formatted at 2026-06-19 10:25:23 on dist-test-slave-xd2q"
server_key: "9df42bdb9d196b8aa490f5eb89bee47b"
server_key_iv: "cdbceb4fe4e97c400340512887c11f9a"
server_key_version: "encryptionkey@0"
I20260619 10:25:23.594869 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "48f521422f554d03b7b595f3b1a62001"
format_stamp: "Formatted at 2026-06-19 10:25:23 on dist-test-slave-xd2q"
server_key: "9df42bdb9d196b8aa490f5eb89bee47b"
server_key_iv: "cdbceb4fe4e97c400340512887c11f9a"
server_key_version: "encryptionkey@0"
I20260619 10:25:23.596189 7394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:23.596884 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:23.597072 7394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:23.597146 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "48f521422f554d03b7b595f3b1a62001"
format_stamp: "Formatted at 2026-06-19 10:25:23 on dist-test-slave-xd2q"
server_key: "9df42bdb9d196b8aa490f5eb89bee47b"
server_key_iv: "cdbceb4fe4e97c400340512887c11f9a"
server_key_version: "encryptionkey@0"
I20260619 10:25:23.597241 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:23.606956 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:23.609314 7394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:23.609459 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:23.613847 7394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33845
I20260619 10:25:23.613919 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33845 every 8 connection(s)
I20260619 10:25:23.614241 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:23.614858 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7394
I20260619 10:25:23.614972 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:23.615168 19718 external_mini_cluster.cc:1428] Setting key b7de01f1b73341a08ebadfc1a394ce51
I20260619 10:25:23.615408 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:23.618669 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001: Bootstrap starting.
I20260619 10:25:23.619366 7462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:23.619702 7462 log.cc:824] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:23.620355 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001: No bootstrap required, opened a new log
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:25:23.622292 7462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260619 10:25:23.622422 7462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:23.622445 7462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f521422f554d03b7b595f3b1a62001, State: Initialized, Role: FOLLOWER
I20260619 10:25:23.622601 7462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260619 10:25:23.622691 7462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:23.622715 7462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:23.622772 7462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:23.623358 7462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260619 10:25:23.623461 7462 leader_election.cc:302] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f521422f554d03b7b595f3b1a62001; no voters:
I20260619 10:25:23.623639 7462 leader_election.cc:288] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:23.623692 7467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:23.623850 7467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [term 1 LEADER]: Becoming Leader. State: Replica: 48f521422f554d03b7b595f3b1a62001, State: Running, Role: LEADER
I20260619 10:25:23.623912 7462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:23.623940 7467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260619 10:25:23.624313 7469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f521422f554d03b7b595f3b1a62001. Latest consensus state: current_term: 1 leader_uuid: "48f521422f554d03b7b595f3b1a62001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260619 10:25:23.624330 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f521422f554d03b7b595f3b1a62001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f521422f554d03b7b595f3b1a62001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260619 10:25:23.624511 7469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:23.624529 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:23.625916 7480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:23.626410 7480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:23.627296 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:23.615789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49739 (local address 127.0.0.1:33845)
0619 10:25:23.615962 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:23.615971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:23.616006 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:25:23.616572 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:23.616578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:23.616623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:23.616736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:23.617527 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:23.618393 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:23.619525 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:23.620007 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:23.620149 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:23.622581 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:23.622585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:23.622586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:23.622597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:23.625596 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:23.626067 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:23.626071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:23.626071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:23.626181 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:23.626456 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:23.626458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:23.626458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:23.626715 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:23.626914 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:23.627004 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 10:25:23.627099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260619 10:25:23.628144 7480 catalog_manager.cc:1347] Generated new cluster ID: 324f999a7df24dfb9b12ba7f2e9a3309
I20260619 10:25:23.628193 7480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:23.628232 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:23.636479 7480 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:23.637238 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:23.629738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35311 (local address 127.0.0.1:33845)
0619 10:25:23.629776 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:23.629778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:23.629910 (+ 132us) server_negotiation.cc:407] Connection header received
0619 10:25:23.629933 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:23.629934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:23.629944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:23.629996 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:23.630588 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:23.631383 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:23.632369 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:23.632689 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:23.632778 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:23.634132 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:23.634133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:23.634134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:23.634140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:23.635852 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:23.636304 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:23.636305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:23.636306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:23.636381 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:23.636654 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:23.636655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:23.636655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:23.636780 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:23.636884 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:23.637068 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 10:25:23.637141 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:23.637284 7480 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:23.646364 7480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f521422f554d03b7b595f3b1a62001: Generated new TSK 0
I20260619 10:25:23.646559 7480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:23.649289 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7394
W20260619 10:25:23.653597 7379 connection.cc:535] client connection to 127.0.0.1:33845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33845): Connection reset by peer (error 104)
2026-06-19T10:25:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7493](info): set up 2 sockets
Jun 19 10:25:23 dist-test-slave-xd2q krb5kdc[7493](info): commencing operation
krb5kdc: starting...
W20260619 10:25:24.704823 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:24 dist-test-slave-xd2q krb5kdc[7493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:24.834934 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:24.917827 7506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:24.918021 7506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:24.918043 7506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:24.919165 7506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:24.919205 7506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:24.919219 7506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:24.919231 7506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:24.919305 7506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:24.920742 7506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:24.921002 7506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:24.921238 7506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:24.926185 7512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:24.926167 7514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:24.926221 7506 server_base.cc:1034] running on GCE node
W20260619 10:25:24.926158 7511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:24.926563 7506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:24.926899 7506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:24.928064 7506 hybrid_clock.cc:648] HybridClock initialized: now 1781864724928064 us; error 40 us; skew 500 ppm
Jun 19 10:25:24 dist-test-slave-xd2q krb5kdc[7493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:24.935608 7506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:24.936121 7506 webserver.cc:460] Webserver started at http://127.0.0.1:38847/ using document root <none> and password file <none>
I20260619 10:25:24.936345 7506 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:24.936422 7506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:24.936573 7506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:24.937637 7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "69d2aaac3c9c40dc9f9568a719e8c08d"
format_stamp: "Formatted at 2026-06-19 10:25:24 on dist-test-slave-xd2q"
server_key: "cc6f0e2d83dda4ea1821749a5e417c21"
server_key_iv: "dbb9308af5fa04fcc600330fb0131413"
server_key_version: "encryptionkey@0"
I20260619 10:25:24.938004 7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "69d2aaac3c9c40dc9f9568a719e8c08d"
format_stamp: "Formatted at 2026-06-19 10:25:24 on dist-test-slave-xd2q"
server_key: "cc6f0e2d83dda4ea1821749a5e417c21"
server_key_iv: "dbb9308af5fa04fcc600330fb0131413"
server_key_version: "encryptionkey@0"
I20260619 10:25:24.939381 7506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 10:25:24.940200 7521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:24.940382 7506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:24.940495 7506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "69d2aaac3c9c40dc9f9568a719e8c08d"
format_stamp: "Formatted at 2026-06-19 10:25:24 on dist-test-slave-xd2q"
server_key: "cc6f0e2d83dda4ea1821749a5e417c21"
server_key_iv: "dbb9308af5fa04fcc600330fb0131413"
server_key_version: "encryptionkey@0"
I20260619 10:25:24.940584 7506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:24.963963 7506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:24.965960 7506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:24.966130 7506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:24.970367 7506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34685
I20260619 10:25:24.970392 7573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34685 every 8 connection(s)
I20260619 10:25:24.970758 7506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:24.971669 7574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:24.973865 7574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d: Bootstrap starting.
I20260619 10:25:24.974395 7574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:24.974622 7574 log.cc:824] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:24.975229 7574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d: No bootstrap required, opened a new log
I20260619 10:25:24.976824 7574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260619 10:25:24.976956 7574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:24.976979 7574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d2aaac3c9c40dc9f9568a719e8c08d, State: Initialized, Role: FOLLOWER
I20260619 10:25:24.977124 7574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260619 10:25:24.977209 7574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:24.977232 7574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:24.977259 7574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:24.977720 7574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260619 10:25:24.977805 7574 leader_election.cc:302] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d2aaac3c9c40dc9f9568a719e8c08d; no voters:
I20260619 10:25:24.977922 7574 leader_election.cc:288] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:24.977982 7577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:24.978152 7577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [term 1 LEADER]: Becoming Leader. State: Replica: 69d2aaac3c9c40dc9f9568a719e8c08d, State: Running, Role: LEADER
I20260619 10:25:24.978168 7574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:24.978242 7577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260619 10:25:24.978626 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d2aaac3c9c40dc9f9568a719e8c08d. Latest consensus state: current_term: 1 leader_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260619 10:25:24.978711 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:24.978842 7578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d2aaac3c9c40dc9f9568a719e8c08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260619 10:25:24.978947 7578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:24.978955 7586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:24.979784 7586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:24.980587 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7506
I20260619 10:25:24.980695 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:24.980872 19718 external_mini_cluster.cc:1428] Setting key e6452407a9f78ec0320b5eb0746b560b
I20260619 10:25:24.981688 7586 catalog_manager.cc:1347] Generated new cluster ID: e5dc251b87fb4aedbd79d169b0aebe36
I20260619 10:25:24.981741 7586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:24.986884 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:24.981541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38545 (local address 127.0.0.1:34685)
0619 10:25:24.981699 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:24.981708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:24.981741 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:25:24.981848 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:24.981855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:24.981907 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:24.982028 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:24.982273 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:24.983651 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:24.984793 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:24.985373 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:24.985474 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:24.986236 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:24.986239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:24.986239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:24.986437 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:24.986470 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:24.986483 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:24.986560 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:24.986679 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 10:25:24.986703 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260619 10:25:24.987921 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:24.997514 7586 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:24.998216 7586 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:25.001227 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:24.997444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46433 (local address 127.0.0.1:34685)
0619 10:25:24.997485 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:24.997487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:24.997674 (+ 187us) server_negotiation.cc:407] Connection header received
0619 10:25:24.997723 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:24.997724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:24.997735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:24.997796 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:24.997968 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:24.998704 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:24.999782 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:25.000087 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:25.000150 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:25.000943 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:25.000945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:25.000945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:25.000949 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:25.000964 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:25.000969 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:25.001027 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:25.001137 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 10:25:25.001149 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":21}
I20260619 10:25:25.002051 7586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d2aaac3c9c40dc9f9568a719e8c08d: Generated new TSK 0
I20260619 10:25:25.002235 7586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:25.007939 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7506
W20260619 10:25:25.011525 7491 connection.cc:535] client connection to 127.0.0.1:34685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34685): Connection reset by peer (error 104)
2026-06-19T10:25:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:25 dist-test-slave-xd2q krb5kdc[7605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:25 dist-test-slave-xd2q krb5kdc[7605](info): set up 2 sockets
Jun 19 10:25:25 dist-test-slave-xd2q krb5kdc[7605](info): commencing operation
krb5kdc: starting...
W20260619 10:25:26.047452 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:26 dist-test-slave-xd2q krb5kdc[7605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:26.164453 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:26.243391 7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:26.243580 7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:26.243604 7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:26.244812 7618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:26.244855 7618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:26.244870 7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:26.244884 7618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:26.244895 7618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:26.246344 7618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:26.246593 7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:26.246824 7618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:26.251675 7626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:26.251835 7618 server_base.cc:1034] running on GCE node
W20260619 10:25:26.251691 7623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:26.251696 7624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:26.252211 7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:26.252521 7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:26.253679 7618 hybrid_clock.cc:648] HybridClock initialized: now 1781864726253676 us; error 36 us; skew 500 ppm
Jun 19 10:25:26 dist-test-slave-xd2q krb5kdc[7605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:26.256261 7618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:26.256670 7618 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260619 10:25:26.256837 7618 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:26.256886 7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:26.256968 7618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:26.257939 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "96ad50fea1a34309bef50b4a06e9c6e6"
format_stamp: "Formatted at 2026-06-19 10:25:26 on dist-test-slave-xd2q"
server_key: "28d50787280cae3cb4b32602ea142a7a"
server_key_iv: "43ec5cb7ec098bd3649f530241e03c8f"
server_key_version: "encryptionkey@0"
I20260619 10:25:26.258222 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "96ad50fea1a34309bef50b4a06e9c6e6"
format_stamp: "Formatted at 2026-06-19 10:25:26 on dist-test-slave-xd2q"
server_key: "28d50787280cae3cb4b32602ea142a7a"
server_key_iv: "43ec5cb7ec098bd3649f530241e03c8f"
server_key_version: "encryptionkey@0"
I20260619 10:25:26.259449 7618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:26.260113 7633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:26.260298 7618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:26.260366 7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "96ad50fea1a34309bef50b4a06e9c6e6"
format_stamp: "Formatted at 2026-06-19 10:25:26 on dist-test-slave-xd2q"
server_key: "28d50787280cae3cb4b32602ea142a7a"
server_key_iv: "43ec5cb7ec098bd3649f530241e03c8f"
server_key_version: "encryptionkey@0"
I20260619 10:25:26.260452 7618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:26.263645 7618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:26.265926 7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:26.266064 7618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:26.269975 7618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39787
I20260619 10:25:26.270107 7685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39787 every 8 connection(s)
I20260619 10:25:26.270365 7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:26.271278 7686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:26.273955 7686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6: Bootstrap starting.
I20260619 10:25:26.274618 7686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:26.274962 7686 log.cc:824] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:26.275802 7686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6: No bootstrap required, opened a new log
I20260619 10:25:26.277611 7686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } }
I20260619 10:25:26.277769 7686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:26.277848 7686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ad50fea1a34309bef50b4a06e9c6e6, State: Initialized, Role: FOLLOWER
I20260619 10:25:26.278065 7686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } }
I20260619 10:25:26.278162 7686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:26.278228 7686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:26.278290 7686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:26.278831 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7618
I20260619 10:25:26.278968 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:26.279065 7686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } }
I20260619 10:25:26.279188 19718 external_mini_cluster.cc:1428] Setting key 02ff2dad022684169e990c28c03e0050
I20260619 10:25:26.279188 7686 leader_election.cc:302] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96ad50fea1a34309bef50b4a06e9c6e6; no voters:
I20260619 10:25:26.279410 7686 leader_election.cc:288] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:26.279441 7689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:26.279594 7689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 96ad50fea1a34309bef50b4a06e9c6e6, State: Running, Role: LEADER
I20260619 10:25:26.279747 7686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:26.279748 7689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } }
I20260619 10:25:26.280227 7691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } } }
I20260619 10:25:26.280354 7691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:26.280376 7690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96ad50fea1a34309bef50b4a06e9c6e6. Latest consensus state: current_term: 1 leader_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ad50fea1a34309bef50b4a06e9c6e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39787 } } }
I20260619 10:25:26.280566 7690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:26.281433 7697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:26.282039 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:26.279726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41551 (local address 127.0.0.1:39787)
0619 10:25:26.280178 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:26.280186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:26.280218 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:26.280331 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:26.280339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:26.280369 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:26.280750 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:26.281523 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:26.281526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:26.281527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:26.281531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:26.281574 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:26.281588 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:26.281653 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:26.281857 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 10:25:26.281893 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":76,"threads_started":1}
I20260619 10:25:26.282173 7697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:26.283118 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:26.283867 7697 catalog_manager.cc:1347] Generated new cluster ID: 93902d6cadaf43b58be915af8887fd6e
I20260619 10:25:26.283917 7697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:26.288757 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:26.284905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44165 (local address 127.0.0.1:39787)
0619 10:25:26.284948 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:26.284951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:26.287596 (+ 2645us) server_negotiation.cc:407] Connection header received
0619 10:25:26.287682 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:26.287683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:26.287689 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:26.287740 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:26.288483 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:26.288485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:26.288486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:26.288490 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:26.288513 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:26.288518 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:26.288591 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:26.288667 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:25:26.288684 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:25:26.291535 7697 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:26.291710 7697 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:26.301095 7697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96ad50fea1a34309bef50b4a06e9c6e6: Generated new TSK 0
I20260619 10:25:26.301319 7697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:26.305457 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7618
2026-06-19T10:25:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:26 dist-test-slave-xd2q krb5kdc[7717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:26 dist-test-slave-xd2q krb5kdc[7717](info): set up 2 sockets
Jun 19 10:25:26 dist-test-slave-xd2q krb5kdc[7717](info): commencing operation
krb5kdc: starting...
W20260619 10:25:27.335235 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:27.450099 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:27.529366 7730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:27.529587 7730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:27.529620 7730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:27.530714 7730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:27.530793 7730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:27.530817 7730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:27.530840 7730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:27.530867 7730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:27.532373 7730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:27.532640 7730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:27.532863 7730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:27.537740 7738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:27.537750 7736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:27.537775 7735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:27.538105 7730 server_base.cc:1034] running on GCE node
I20260619 10:25:27.538367 7730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:27.538652 7730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:27.539819 7730 hybrid_clock.cc:648] HybridClock initialized: now 1781864727539813 us; error 43 us; skew 500 ppm
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:27.542250 7730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:27.542619 7730 webserver.cc:460] Webserver started at http://127.0.0.1:39163/ using document root <none> and password file <none>
I20260619 10:25:27.542838 7730 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:27.542889 7730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:27.543066 7730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:27.544144 7730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "c20bff43655249f8ade2dfa67d779c32"
format_stamp: "Formatted at 2026-06-19 10:25:27 on dist-test-slave-xd2q"
server_key: "b51725d9414a72ba9aa2c9e8808d9b93"
server_key_iv: "568d993e08c029a9b22263676de1eb61"
server_key_version: "encryptionkey@0"
I20260619 10:25:27.544514 7730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "c20bff43655249f8ade2dfa67d779c32"
format_stamp: "Formatted at 2026-06-19 10:25:27 on dist-test-slave-xd2q"
server_key: "b51725d9414a72ba9aa2c9e8808d9b93"
server_key_iv: "568d993e08c029a9b22263676de1eb61"
server_key_version: "encryptionkey@0"
I20260619 10:25:27.545817 7730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:25:27.546517 7745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:27.546686 7730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:27.546787 7730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "c20bff43655249f8ade2dfa67d779c32"
format_stamp: "Formatted at 2026-06-19 10:25:27 on dist-test-slave-xd2q"
server_key: "b51725d9414a72ba9aa2c9e8808d9b93"
server_key_iv: "568d993e08c029a9b22263676de1eb61"
server_key_version: "encryptionkey@0"
I20260619 10:25:27.546866 7730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:27.555390 7730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:27.557514 7730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:27.557685 7730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:27.561424 7730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40755
I20260619 10:25:27.561452 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40755 every 8 connection(s)
I20260619 10:25:27.561760 7730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:27.562587 7798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:27.564632 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7730
I20260619 10:25:27.564765 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:27.564790 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32: Bootstrap starting.
I20260619 10:25:27.564946 19718 external_mini_cluster.cc:1428] Setting key 9f3d0ff36b605890b088e3c2aaa7b1b9
I20260619 10:25:27.565488 7798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:27.565821 7798 log.cc:824] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:27.566526 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32: No bootstrap required, opened a new log
I20260619 10:25:27.568862 7798 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260619 10:25:27.569051 7798 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:27.569100 7798 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20bff43655249f8ade2dfa67d779c32, State: Initialized, Role: FOLLOWER
I20260619 10:25:27.569218 7798 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260619 10:25:27.569295 7798 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:27.569330 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:27.569384 7798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:27.570050 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260619 10:25:27.570168 7798 leader_election.cc:302] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20bff43655249f8ade2dfa67d779c32; no voters:
I20260619 10:25:27.570307 7798 leader_election.cc:288] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:27.570374 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:27.570542 7798 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:27.570569 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [term 1 LEADER]: Becoming Leader. State: Replica: c20bff43655249f8ade2dfa67d779c32, State: Running, Role: LEADER
I20260619 10:25:27.570755 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260619 10:25:27.571153 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20bff43655249f8ade2dfa67d779c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } } }
I20260619 10:25:27.571234 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:27.571564 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20bff43655249f8ade2dfa67d779c32. Latest consensus state: current_term: 1 leader_uuid: "c20bff43655249f8ade2dfa67d779c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bff43655249f8ade2dfa67d779c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } } }
I20260619 10:25:27.571692 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:27.571848 7811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:27.572301 7811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:25:27.574595 7811 catalog_manager.cc:1347] Generated new cluster ID: eef1f11e0fe54aa4956bd0212fc413bb
I20260619 10:25:27.574654 7811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:27.576740 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:27.565506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57025 (local address 127.0.0.1:40755)
0619 10:25:27.565693 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:27.565701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:27.565733 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:27.566294 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:27.566303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:27.566361 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:27.566508 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:27.567233 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:27.568709 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:27.569754 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:27.570410 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:27.570811 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:27.573357 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:27.573361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:27.573362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:27.573373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:27.575192 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:27.575593 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:27.575596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:27.575597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:27.575685 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:27.575894 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:27.575895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:27.575895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:27.576110 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:27.576295 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:27.576388 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 10:25:27.576530 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260619 10:25:27.577649 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:27.583858 7811 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:27.584520 7811 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:27.585107 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:27.578935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46727 (local address 127.0.0.1:40755)
0619 10:25:27.578971 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:27.578972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:27.579091 (+ 119us) server_negotiation.cc:407] Connection header received
0619 10:25:27.579160 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:27.579161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:27.579171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:27.579225 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:27.579755 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:27.580231 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:27.580956 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:27.581170 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:27.581243 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:27.582389 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:27.582391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:27.582391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:27.582397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:27.583983 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:27.584320 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:27.584321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:27.584321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:27.584371 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:27.584577 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:27.584578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:27.584578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:27.584683 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:27.584782 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:27.584902 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:25:27.584994 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:25:27.592738 7811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20bff43655249f8ade2dfa67d779c32: Generated new TSK 0
I20260619 10:25:27.592937 7811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:27.596063 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7730
2026-06-19T10:25:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7829](info): set up 2 sockets
Jun 19 10:25:27 dist-test-slave-xd2q krb5kdc[7829](info): commencing operation
krb5kdc: starting...
W20260619 10:25:28.615471 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:28 dist-test-slave-xd2q krb5kdc[7829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:28.728109 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:28.808166 7842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:28.808347 7842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:28.808367 7842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:28.809516 7842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:28.809573 7842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:28.809587 7842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:28.809599 7842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:28.809612 7842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:28.811098 7842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:28.811378 7842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:28.811602 7842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:28.816635 7848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:28.816872 7850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:28.816898 7842 server_base.cc:1034] running on GCE node
W20260619 10:25:28.816665 7847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:28.817219 7842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:28.817519 7842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:28.818687 7842 hybrid_clock.cc:648] HybridClock initialized: now 1781864728818670 us; error 36 us; skew 500 ppm
Jun 19 10:25:28 dist-test-slave-xd2q krb5kdc[7829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:28.821189 7842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:28.821640 7842 webserver.cc:460] Webserver started at http://127.0.0.1:42337/ using document root <none> and password file <none>
I20260619 10:25:28.821834 7842 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:28.821874 7842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:28.821998 7842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:28.822979 7842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "69d521f061374561b23630c3e4bf64f7"
format_stamp: "Formatted at 2026-06-19 10:25:28 on dist-test-slave-xd2q"
server_key: "f21ddde971ac6d324be96b21eeef699e"
server_key_iv: "4f1f7fe2b6d3c832fcbc12f4a10b5c26"
server_key_version: "encryptionkey@0"
I20260619 10:25:28.823275 7842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "69d521f061374561b23630c3e4bf64f7"
format_stamp: "Formatted at 2026-06-19 10:25:28 on dist-test-slave-xd2q"
server_key: "f21ddde971ac6d324be96b21eeef699e"
server_key_iv: "4f1f7fe2b6d3c832fcbc12f4a10b5c26"
server_key_version: "encryptionkey@0"
I20260619 10:25:28.824576 7842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:28.825237 7857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:28.825408 7842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:28.825477 7842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "69d521f061374561b23630c3e4bf64f7"
format_stamp: "Formatted at 2026-06-19 10:25:28 on dist-test-slave-xd2q"
server_key: "f21ddde971ac6d324be96b21eeef699e"
server_key_iv: "4f1f7fe2b6d3c832fcbc12f4a10b5c26"
server_key_version: "encryptionkey@0"
I20260619 10:25:28.825524 7842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:28.841174 7842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:28.842932 7842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:28.843039 7842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:28.847002 7842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44467
I20260619 10:25:28.847064 7909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44467 every 8 connection(s)
I20260619 10:25:28.847453 7842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:28.848307 7910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:28.850355 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7: Bootstrap starting.
I20260619 10:25:28.850833 7910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:28.851053 7910 log.cc:824] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:28.851729 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7: No bootstrap required, opened a new log
I20260619 10:25:28.852442 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7842
I20260619 10:25:28.852550 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:28.852753 19718 external_mini_cluster.cc:1428] Setting key d837f7c35b86471861c3410bc4c543b4
I20260619 10:25:28.853785 7910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } }
I20260619 10:25:28.853952 7910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:28.853989 7910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d521f061374561b23630c3e4bf64f7, State: Initialized, Role: FOLLOWER
I20260619 10:25:28.854146 7910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } }
I20260619 10:25:28.854235 7910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:28.854290 7910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:28.854347 7910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:28.854897 7910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } }
I20260619 10:25:28.854979 7910 leader_election.cc:302] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d521f061374561b23630c3e4bf64f7; no voters:
I20260619 10:25:28.855154 7910 leader_election.cc:288] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:28.855175 7915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:28.855278 7915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [term 1 LEADER]: Becoming Leader. State: Replica: 69d521f061374561b23630c3e4bf64f7, State: Running, Role: LEADER
I20260619 10:25:28.855427 7910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:28.855434 7915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } }
I20260619 10:25:28.855846 7915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d521f061374561b23630c3e4bf64f7. Latest consensus state: current_term: 1 leader_uuid: "69d521f061374561b23630c3e4bf64f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } } }
I20260619 10:25:28.855857 7917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d521f061374561b23630c3e4bf64f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d521f061374561b23630c3e4bf64f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44467 } } }
I20260619 10:25:28.856015 7915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:28.856029 7917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:28.856292 7924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:28.856848 7924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:28.858855 7924 catalog_manager.cc:1347] Generated new cluster ID: fec03ef81a5e44a095408d3a5a248833
I20260619 10:25:28.858899 7924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:28.858904 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:28.853270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41055 (local address 127.0.0.1:44467)
0619 10:25:28.853470 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:28.853477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:28.853504 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:25:28.853777 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:28.853783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:28.853832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:28.853943 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:28.854218 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:28.855675 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:28.856858 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:28.857382 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:28.857505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:28.858137 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:28.858141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:28.858141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:28.858345 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:28.858380 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:28.858393 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:28.858477 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:28.858646 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 10:25:28.858703 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260619 10:25:28.863903 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:28.866191 7924 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:28.866885 7924 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:28.868458 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:28.865240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46729 (local address 127.0.0.1:44467)
0619 10:25:28.865278 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:28.865279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:28.865386 (+ 107us) server_negotiation.cc:407] Connection header received
0619 10:25:28.865452 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:28.865453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:28.865463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:28.865515 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:28.865645 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:28.866167 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:28.867178 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:28.867495 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:28.867529 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:28.868234 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:28.868236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:28.868236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:28.868240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:28.868251 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:28.868254 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:28.868277 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:28.868358 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 10:25:28.868392 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:28.873881 7924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d521f061374561b23630c3e4bf64f7: Generated new TSK 0
I20260619 10:25:28.874047 7924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:28.875916 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7842
2026-06-19T10:25:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:28 dist-test-slave-xd2q krb5kdc[7941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:28 dist-test-slave-xd2q krb5kdc[7941](info): set up 2 sockets
Jun 19 10:25:28 dist-test-slave-xd2q krb5kdc[7941](info): commencing operation
krb5kdc: starting...
W20260619 10:25:29.907398 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:29 dist-test-slave-xd2q krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:30.032148 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:30.117815 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:30.118050 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:30.118088 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:30.119249 7954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:30.119351 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:30.119386 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:30.119405 7954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:30.119427 7954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:30.120896 7954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:30.121161 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:30.121400 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:30.126791 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:30.126776 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:30.126776 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:30.127091 7954 server_base.cc:1034] running on GCE node
I20260619 10:25:30.127391 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:30.127691 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:30.128876 7954 hybrid_clock.cc:648] HybridClock initialized: now 1781864730128864 us; error 48 us; skew 500 ppm
Jun 19 10:25:30 dist-test-slave-xd2q krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:30.135995 7954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:30.137241 7954 webserver.cc:460] Webserver started at http://127.0.0.1:41885/ using document root <none> and password file <none>
I20260619 10:25:30.137605 7954 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:30.137682 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:30.137822 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:30.138969 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "41fa02d8366642069855e844659e9ec5"
format_stamp: "Formatted at 2026-06-19 10:25:30 on dist-test-slave-xd2q"
server_key: "0dee12fcc2d977fbc29833c110cb7848"
server_key_iv: "6cc0cac4a5f37c875b56bf99e7902b15"
server_key_version: "encryptionkey@0"
I20260619 10:25:30.139429 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "41fa02d8366642069855e844659e9ec5"
format_stamp: "Formatted at 2026-06-19 10:25:30 on dist-test-slave-xd2q"
server_key: "0dee12fcc2d977fbc29833c110cb7848"
server_key_iv: "6cc0cac4a5f37c875b56bf99e7902b15"
server_key_version: "encryptionkey@0"
I20260619 10:25:30.141268 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:30.142287 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:30.142467 7954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:30.142581 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "41fa02d8366642069855e844659e9ec5"
format_stamp: "Formatted at 2026-06-19 10:25:30 on dist-test-slave-xd2q"
server_key: "0dee12fcc2d977fbc29833c110cb7848"
server_key_iv: "6cc0cac4a5f37c875b56bf99e7902b15"
server_key_version: "encryptionkey@0"
I20260619 10:25:30.142690 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:30.148505 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:30.151611 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:30.151821 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:30.157336 7954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33461
I20260619 10:25:30.157363 8021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33461 every 8 connection(s)
I20260619 10:25:30.157920 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:30.159111 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:30.162948 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5: Bootstrap starting.
I20260619 10:25:30.164024 8022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:30.164404 8022 log.cc:824] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:30.165211 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5: No bootstrap required, opened a new log
I20260619 10:25:30.167956 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 7954
I20260619 10:25:30.168087 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:30.168078 8022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260619 10:25:30.168298 8022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:30.168316 19718 external_mini_cluster.cc:1428] Setting key 27c438d6e8f35dd1e8b219eb3ae15262
I20260619 10:25:30.168361 8022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fa02d8366642069855e844659e9ec5, State: Initialized, Role: FOLLOWER
I20260619 10:25:30.168610 8022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260619 10:25:30.168756 8022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:30.168800 8022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:30.168849 8022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:30.169708 8022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260619 10:25:30.169863 8022 leader_election.cc:302] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41fa02d8366642069855e844659e9ec5; no voters:
I20260619 10:25:30.170071 8022 leader_election.cc:288] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:30.170107 8027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:30.170238 8027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 41fa02d8366642069855e844659e9ec5, State: Running, Role: LEADER
I20260619 10:25:30.170341 8022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:30.170339 8027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260619 10:25:30.170759 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41fa02d8366642069855e844659e9ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } } }
I20260619 10:25:30.170862 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:30.171026 8029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41fa02d8366642069855e844659e9ec5. Latest consensus state: current_term: 1 leader_uuid: "41fa02d8366642069855e844659e9ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fa02d8366642069855e844659e9ec5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } } }
I20260619 10:25:30.171110 8029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:30.171237 8033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:30.172034 8033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:30.173657 8033 catalog_manager.cc:1347] Generated new cluster ID: 17a71bd8ff93457c8a08a1c2d7caa5a6
I20260619 10:25:30.173749 8033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:30.176875 8033 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:30.177014 8033 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:30.180531 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:30.168984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40909 (local address 127.0.0.1:33461)
0619 10:25:30.169188 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:30.169199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:30.169235 (+ 36us) server_negotiation.cc:407] Connection header received
0619 10:25:30.169659 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:30.169669 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:30.169709 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:30.170015 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:30.179973 (+ 9958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:30.179978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:30.179978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:30.179984 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:30.180036 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:30.180053 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:30.180109 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:30.180283 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 10:25:30.180330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260619 10:25:30.181423 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:30.184367 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:30.183168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56419 (local address 127.0.0.1:33461)
0619 10:25:30.183211 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:30.183213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:30.183325 (+ 112us) server_negotiation.cc:407] Connection header received
0619 10:25:30.183424 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:30.183425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:30.183431 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:30.183456 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:30.184137 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:30.184138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:30.184139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:30.184142 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:30.184155 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:30.184159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:30.184185 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:30.184270 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 10:25:30.184283 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:25:30.186617 8033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41fa02d8366642069855e844659e9ec5: Generated new TSK 0
I20260619 10:25:30.186820 8033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:30.192576 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 7954
2026-06-19T10:25:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:30 dist-test-slave-xd2q krb5kdc[8053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:30 dist-test-slave-xd2q krb5kdc[8053](info): set up 2 sockets
Jun 19 10:25:30 dist-test-slave-xd2q krb5kdc[8053](info): commencing operation
krb5kdc: starting...
W20260619 10:25:31.235832 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:31.353609 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:31.432152 8066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:31.432328 8066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:31.432348 8066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:31.433542 8066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:31.433584 8066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:31.433598 8066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:31.433610 8066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:31.433624 8066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:31.435091 8066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:31.435361 8066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:31.435585 8066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:31.440222 8074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:31.440266 8072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:31.440263 8071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:31.440445 8066 server_base.cc:1034] running on GCE node
I20260619 10:25:31.440683 8066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:31.440984 8066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:31.442173 8066 hybrid_clock.cc:648] HybridClock initialized: now 1781864731442165 us; error 41 us; skew 500 ppm
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:31.444868 8066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:31.445309 8066 webserver.cc:460] Webserver started at http://127.0.0.1:43691/ using document root <none> and password file <none>
I20260619 10:25:31.445549 8066 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:31.445621 8066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:31.445751 8066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:31.446801 8066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "6d525742eb324d8197d91d5972d5650c"
format_stamp: "Formatted at 2026-06-19 10:25:31 on dist-test-slave-xd2q"
server_key: "1806b4a40752d43fa584049be1fe1e2e"
server_key_iv: "031f8d5cd06839f190744e82ece8872b"
server_key_version: "encryptionkey@0"
I20260619 10:25:31.447168 8066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "6d525742eb324d8197d91d5972d5650c"
format_stamp: "Formatted at 2026-06-19 10:25:31 on dist-test-slave-xd2q"
server_key: "1806b4a40752d43fa584049be1fe1e2e"
server_key_iv: "031f8d5cd06839f190744e82ece8872b"
server_key_version: "encryptionkey@0"
I20260619 10:25:31.448565 8066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:31.449298 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:31.449468 8066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:31.449569 8066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "6d525742eb324d8197d91d5972d5650c"
format_stamp: "Formatted at 2026-06-19 10:25:31 on dist-test-slave-xd2q"
server_key: "1806b4a40752d43fa584049be1fe1e2e"
server_key_iv: "031f8d5cd06839f190744e82ece8872b"
server_key_version: "encryptionkey@0"
I20260619 10:25:31.449653 8066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:31.465066 8066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:31.467343 8066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:31.467504 8066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:31.471195 8066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35989
I20260619 10:25:31.471240 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35989 every 8 connection(s)
I20260619 10:25:31.471580 8066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:31.472375 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:31.474502 8134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c: Bootstrap starting.
I20260619 10:25:31.475063 8134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:31.475409 8134 log.cc:824] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:31.476070 8134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c: No bootstrap required, opened a new log
I20260619 10:25:31.477649 8134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } }
I20260619 10:25:31.477782 8134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:31.477803 8134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d525742eb324d8197d91d5972d5650c, State: Initialized, Role: FOLLOWER
I20260619 10:25:31.477984 8134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } }
I20260619 10:25:31.478091 8134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:31.478089 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8066
I20260619 10:25:31.478150 8134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:31.478225 8134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:31.478230 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:31.478443 19718 external_mini_cluster.cc:1428] Setting key 322c9e8e2d78fe158fae2eb1cbd43404
I20260619 10:25:31.478914 8134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } }
I20260619 10:25:31.479025 8134 leader_election.cc:302] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d525742eb324d8197d91d5972d5650c; no voters:
I20260619 10:25:31.479212 8134 leader_election.cc:288] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:31.479244 8139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:31.479403 8139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [term 1 LEADER]: Becoming Leader. State: Replica: 6d525742eb324d8197d91d5972d5650c, State: Running, Role: LEADER
I20260619 10:25:31.479482 8134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:31.479493 8139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } }
I20260619 10:25:31.480000 8141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d525742eb324d8197d91d5972d5650c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } } }
I20260619 10:25:31.480069 8140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d525742eb324d8197d91d5972d5650c. Latest consensus state: current_term: 1 leader_uuid: "6d525742eb324d8197d91d5972d5650c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d525742eb324d8197d91d5972d5650c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35989 } } }
I20260619 10:25:31.480151 8140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:31.480470 8148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:31.480140 8141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:31.481042 8148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:31.482947 8148 catalog_manager.cc:1347] Generated new cluster ID: 26ae5e412f1546ecbdeaa29bcc2c1dac
I20260619 10:25:31.483022 8148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:25:31.490798 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:31.478923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:57307 (local address 127.0.0.1:35989)
0619 10:25:31.479108 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:31.479117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:31.479152 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:25:31.479270 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:31.479278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:31.479363 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:31.480345 (+ 982us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:31.480443 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:31.481927 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:31.482932 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:31.483572 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:31.483696 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:31.486488 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:31.486491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:31.486492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:31.486503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:31.489021 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:31.489519 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:31.489522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:31.489522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:31.489637 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:31.489909 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:31.489910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:31.489910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:31.490196 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:31.490343 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:31.490495 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 10:25:31.490590 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260619 10:25:31.491942 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:31.500581 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:31.493502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:34349 (local address 127.0.0.1:35989)
0619 10:25:31.493537 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:31.493538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:31.493549 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:25:31.493586 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:31.493587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:31.493596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:31.493654 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:31.494216 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:31.494983 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:31.495978 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:31.496309 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:31.496382 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:31.497725 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:31.497726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:31.497727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:31.497734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:31.499379 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:31.499780 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:31.499781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:31.499781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:31.499842 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:31.500072 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:31.500073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:31.500073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:31.500197 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:31.500323 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:31.500424 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:25:31.500486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:31.505175 8148 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:31.505702 8148 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:31.513607 8148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d525742eb324d8197d91d5972d5650c: Generated new TSK 0
I20260619 10:25:31.513794 8148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:31.520464 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8066
W20260619 10:25:31.524744 8051 connection.cc:535] client connection to 127.0.0.1:35989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35989): Connection reset by peer (error 104)
2026-06-19T10:25:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8165](info): set up 2 sockets
Jun 19 10:25:31 dist-test-slave-xd2q krb5kdc[8165](info): commencing operation
krb5kdc: starting...
W20260619 10:25:32.551381 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:32 dist-test-slave-xd2q krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:32.667923 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:32.748849 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:32.749094 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:32.749130 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:32.750227 8178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:32.750289 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:32.750313 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:32.750342 8178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:32.750370 8178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:32.751894 8178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:32.752151 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:32.752388 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:32.757149 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:32.757162 8183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:32.757164 8184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:32.757467 8178 server_base.cc:1034] running on GCE node
I20260619 10:25:32.757711 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:32.757980 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:32.759142 8178 hybrid_clock.cc:648] HybridClock initialized: now 1781864732759133 us; error 35 us; skew 500 ppm
Jun 19 10:25:32 dist-test-slave-xd2q krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:32.761680 8178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:32.762169 8178 webserver.cc:460] Webserver started at http://127.0.0.1:43255/ using document root <none> and password file <none>
I20260619 10:25:32.762395 8178 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:32.762457 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:32.762595 8178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:32.763710 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "52cfa5567fef4b5c8cccc231aa34d7aa"
format_stamp: "Formatted at 2026-06-19 10:25:32 on dist-test-slave-xd2q"
server_key: "5b21e8987f249aa5c446ae91a3686efc"
server_key_iv: "ab65ea61df26914d5c45aba53c4712e4"
server_key_version: "encryptionkey@0"
I20260619 10:25:32.764042 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "52cfa5567fef4b5c8cccc231aa34d7aa"
format_stamp: "Formatted at 2026-06-19 10:25:32 on dist-test-slave-xd2q"
server_key: "5b21e8987f249aa5c446ae91a3686efc"
server_key_iv: "ab65ea61df26914d5c45aba53c4712e4"
server_key_version: "encryptionkey@0"
I20260619 10:25:32.765300 8178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 10:25:32.766052 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:32.766232 8178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:32.766331 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "52cfa5567fef4b5c8cccc231aa34d7aa"
format_stamp: "Formatted at 2026-06-19 10:25:32 on dist-test-slave-xd2q"
server_key: "5b21e8987f249aa5c446ae91a3686efc"
server_key_iv: "ab65ea61df26914d5c45aba53c4712e4"
server_key_version: "encryptionkey@0"
I20260619 10:25:32.766399 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:32.782582 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:32.784440 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:32.784598 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:32.788668 8178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38309
I20260619 10:25:32.788702 8245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38309 every 8 connection(s)
I20260619 10:25:32.789069 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:32.789927 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:32.792164 8246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa: Bootstrap starting.
I20260619 10:25:32.792443 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8178
I20260619 10:25:32.792567 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:32.792742 19718 external_mini_cluster.cc:1428] Setting key 710bc2b2550eb08fee6c84bb894244d6
I20260619 10:25:32.792784 8246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:32.793087 8246 log.cc:824] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:32.793846 8246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa: No bootstrap required, opened a new log
I20260619 10:25:32.796036 8246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260619 10:25:32.796197 8246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:32.796231 8246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52cfa5567fef4b5c8cccc231aa34d7aa, State: Initialized, Role: FOLLOWER
I20260619 10:25:32.796346 8246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260619 10:25:32.796425 8246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:32.796476 8246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:32.796525 8246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:32.797174 8246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260619 10:25:32.797302 8246 leader_election.cc:302] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52cfa5567fef4b5c8cccc231aa34d7aa; no voters:
I20260619 10:25:32.797477 8246 leader_election.cc:288] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:32.797577 8251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:32.797752 8246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:32.797756 8251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 52cfa5567fef4b5c8cccc231aa34d7aa, State: Running, Role: LEADER
I20260619 10:25:32.797870 8251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260619 10:25:32.798169 8252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } } }
I20260619 10:25:32.798256 8252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:32.798492 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52cfa5567fef4b5c8cccc231aa34d7aa. Latest consensus state: current_term: 1 leader_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cfa5567fef4b5c8cccc231aa34d7aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } } }
I20260619 10:25:32.798548 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:32.793226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:58445 (local address 127.0.0.1:38309)
0619 10:25:32.793387 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:32.793396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:32.793442 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:25:32.793901 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:32.793909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:32.793970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:32.794091 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:32.794330 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:32.795182 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:32.796346 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:32.796840 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:32.796920 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:32.797749 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:32.797753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:32.797754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:32.797964 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:32.798003 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:32.798017 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:32.798075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:32.798268 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 10:25:32.798299 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260619 10:25:32.798862 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:32.799023 8258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:32.799548 8258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:32.799619 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:32.803714 8258 catalog_manager.cc:1347] Generated new cluster ID: 2bfa27d2eefd49d7b1d9acef560a2bd5
I20260619 10:25:32.803766 8258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:32.807574 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:32.803911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:41225 (local address 127.0.0.1:38309)
0619 10:25:32.803948 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:32.803950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:32.804126 (+ 176us) server_negotiation.cc:407] Connection header received
0619 10:25:32.804233 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:32.804234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:32.804243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:32.804271 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:32.804459 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:32.805189 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:32.806268 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:32.806578 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:32.806642 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:32.807330 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:32.807332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:32.807332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:32.807335 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:32.807348 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:32.807352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:32.807408 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:32.807500 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 10:25:32.807509 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:32.826061 8258 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:32.826790 8258 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:32.833092 8258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52cfa5567fef4b5c8cccc231aa34d7aa: Generated new TSK 0
I20260619 10:25:32.833287 8258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:32.843147 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8178
W20260619 10:25:32.847426 8163 connection.cc:535] client connection to 127.0.0.1:38309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38309): Connection reset by peer (error 104)
2026-06-19T10:25:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:32 dist-test-slave-xd2q krb5kdc[8277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:32 dist-test-slave-xd2q krb5kdc[8277](info): set up 2 sockets
Jun 19 10:25:32 dist-test-slave-xd2q krb5kdc[8277](info): commencing operation
krb5kdc: starting...
W20260619 10:25:33.871219 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:33 dist-test-slave-xd2q krb5kdc[8277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:33.984223 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:34.062405 8290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:34.062624 8290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:34.062660 8290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:34.063910 8290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:34.063974 8290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:34.063999 8290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:34.064026 8290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:34.064056 8290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:34.065474 8290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:34.065738 8290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:34.065968 8290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:34.070675 8295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:34.070775 8290 server_base.cc:1034] running on GCE node
W20260619 10:25:34.070753 8296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:34.070755 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:34.071143 8290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:34.071504 8290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:34.072713 8290 hybrid_clock.cc:648] HybridClock initialized: now 1781864734072686 us; error 46 us; skew 500 ppm
Jun 19 10:25:34 dist-test-slave-xd2q krb5kdc[8277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:34.075145 8290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:34.075624 8290 webserver.cc:460] Webserver started at http://127.0.0.1:39921/ using document root <none> and password file <none>
I20260619 10:25:34.075866 8290 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:34.075947 8290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:34.076100 8290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:34.077327 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "ed47aaa8e5c34c83b7243d3e20b42753"
format_stamp: "Formatted at 2026-06-19 10:25:34 on dist-test-slave-xd2q"
server_key: "71946376b36e996b662c6a2ccdc965cd"
server_key_iv: "449cfecf55a21348be47e676adedae94"
server_key_version: "encryptionkey@0"
I20260619 10:25:34.077754 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "ed47aaa8e5c34c83b7243d3e20b42753"
format_stamp: "Formatted at 2026-06-19 10:25:34 on dist-test-slave-xd2q"
server_key: "71946376b36e996b662c6a2ccdc965cd"
server_key_iv: "449cfecf55a21348be47e676adedae94"
server_key_version: "encryptionkey@0"
I20260619 10:25:34.079017 8290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:34.079838 8305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:34.080027 8290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:25:34.080117 8290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "ed47aaa8e5c34c83b7243d3e20b42753"
format_stamp: "Formatted at 2026-06-19 10:25:34 on dist-test-slave-xd2q"
server_key: "71946376b36e996b662c6a2ccdc965cd"
server_key_iv: "449cfecf55a21348be47e676adedae94"
server_key_version: "encryptionkey@0"
I20260619 10:25:34.080183 8290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:34.083389 8290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:34.085682 8290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:34.085860 8290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:34.089905 8290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45239
I20260619 10:25:34.089939 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45239 every 8 connection(s)
I20260619 10:25:34.090250 8290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:34.091095 8358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:34.093209 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753: Bootstrap starting.
I20260619 10:25:34.093710 8358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:34.093945 8358 log.cc:824] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:34.094555 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753: No bootstrap required, opened a new log
I20260619 10:25:34.096171 8358 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260619 10:25:34.096297 8358 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:34.096320 8358 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47aaa8e5c34c83b7243d3e20b42753, State: Initialized, Role: FOLLOWER
I20260619 10:25:34.096463 8358 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260619 10:25:34.096537 8358 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:34.096558 8358 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:34.096585 8358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:34.097062 8358 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260619 10:25:34.097141 8358 leader_election.cc:302] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed47aaa8e5c34c83b7243d3e20b42753; no voters:
I20260619 10:25:34.097247 8358 leader_election.cc:288] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:34.097328 8361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:34.097492 8361 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [term 1 LEADER]: Becoming Leader. State: Replica: ed47aaa8e5c34c83b7243d3e20b42753, State: Running, Role: LEADER
I20260619 10:25:34.097525 8358 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:34.097580 8361 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260619 10:25:34.097970 8363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed47aaa8e5c34c83b7243d3e20b42753. Latest consensus state: current_term: 1 leader_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260619 10:25:34.097934 8362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47aaa8e5c34c83b7243d3e20b42753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260619 10:25:34.098047 8363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:34.098053 8362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:34.098245 8365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:34.098353 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8290
I20260619 10:25:34.098461 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:34.098623 19718 external_mini_cluster.cc:1428] Setting key 5bbe495c9944b3414c064006e7e34fe7
I20260619 10:25:34.098747 8365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:34.100347 8365 catalog_manager.cc:1347] Generated new cluster ID: 4d777580fbdb463fa51f3315ab863a2d
I20260619 10:25:34.100404 8365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:34.100998 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:34.099171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:45467 (local address 127.0.0.1:45239)
0619 10:25:34.099464 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:34.099473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:34.099505 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:34.099609 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:34.099615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:34.099643 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:34.099788 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:34.100531 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:34.100535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:34.100535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:34.100539 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:34.100577 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:34.100593 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:34.100652 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:34.100777 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 10:25:34.100822 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260619 10:25:34.101873 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:34.104357 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:34.103377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:56891 (local address 127.0.0.1:45239)
0619 10:25:34.103408 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:34.103410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:34.103515 (+ 105us) server_negotiation.cc:407] Connection header received
0619 10:25:34.103542 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:34.103543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:34.103547 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:34.103612 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:34.104139 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:34.104140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:34.104140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:34.104144 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:34.104159 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:34.104164 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:34.104226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:34.104284 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 10:25:34.104297 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:25:34.110699 8365 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:34.110857 8365 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:34.116925 8365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed47aaa8e5c34c83b7243d3e20b42753: Generated new TSK 0
I20260619 10:25:34.117101 8365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:34.122563 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8290
2026-06-19T10:25:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:34 dist-test-slave-xd2q krb5kdc[8389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:34 dist-test-slave-xd2q krb5kdc[8389](info): set up 2 sockets
Jun 19 10:25:34 dist-test-slave-xd2q krb5kdc[8389](info): commencing operation
krb5kdc: starting...
W20260619 10:25:35.163479 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:35.280215 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:35.361860 8402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:35.362063 8402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:35.362088 8402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:35.363366 8402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:35.363422 8402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:35.363440 8402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:35.363454 8402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:35.363504 8402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:35.365067 8402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:35.365321 8402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:35.365576 8402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:35.370402 8408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:35.370476 8410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:35.370402 8407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:35.370605 8402 server_base.cc:1034] running on GCE node
I20260619 10:25:35.370832 8402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:35.371135 8402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:35.372290 8402 hybrid_clock.cc:648] HybridClock initialized: now 1781864735372247 us; error 54 us; skew 500 ppm
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:35.380329 8402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:35.380757 8402 webserver.cc:460] Webserver started at http://127.0.0.1:37367/ using document root <none> and password file <none>
I20260619 10:25:35.380928 8402 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:35.380966 8402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:35.381052 8402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:35.382037 8402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "487f479908bf40629dea94766fce1a22"
format_stamp: "Formatted at 2026-06-19 10:25:35 on dist-test-slave-xd2q"
server_key: "30975ed3792a58a0fa29a3fe0ead424c"
server_key_iv: "fa5da051afd05e73fa5ad6c5431728d9"
server_key_version: "encryptionkey@0"
I20260619 10:25:35.382330 8402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "487f479908bf40629dea94766fce1a22"
format_stamp: "Formatted at 2026-06-19 10:25:35 on dist-test-slave-xd2q"
server_key: "30975ed3792a58a0fa29a3fe0ead424c"
server_key_iv: "fa5da051afd05e73fa5ad6c5431728d9"
server_key_version: "encryptionkey@0"
I20260619 10:25:35.383559 8402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:35.384225 8417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:35.384433 8402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:35.384514 8402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "487f479908bf40629dea94766fce1a22"
format_stamp: "Formatted at 2026-06-19 10:25:35 on dist-test-slave-xd2q"
server_key: "30975ed3792a58a0fa29a3fe0ead424c"
server_key_iv: "fa5da051afd05e73fa5ad6c5431728d9"
server_key_version: "encryptionkey@0"
I20260619 10:25:35.384567 8402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:35.396253 8402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:35.398048 8402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:35.398149 8402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:35.401820 8402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46397
I20260619 10:25:35.402103 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46397 every 8 connection(s)
I20260619 10:25:35.402150 8402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:35.402822 8470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:35.404773 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8402
I20260619 10:25:35.404897 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:35.405084 19718 external_mini_cluster.cc:1428] Setting key 1abd74f95300728ad00389d424876866
I20260619 10:25:35.405478 8470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: Bootstrap starting.
I20260619 10:25:35.406131 8470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:35.406450 8470 log.cc:824] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:35.407364 8470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: No bootstrap required, opened a new log
I20260619 10:25:35.409765 8470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260619 10:25:35.409935 8470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:35.409983 8470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487f479908bf40629dea94766fce1a22, State: Initialized, Role: FOLLOWER
I20260619 10:25:35.410123 8470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260619 10:25:35.410233 8470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:35.410269 8470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:35.410362 8470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:35.411083 8470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260619 10:25:35.411217 8470 leader_election.cc:302] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 487f479908bf40629dea94766fce1a22; no voters:
I20260619 10:25:35.411473 8470 leader_election.cc:288] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:35.411494 8475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:35.411609 8475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [term 1 LEADER]: Becoming Leader. State: Replica: 487f479908bf40629dea94766fce1a22, State: Running, Role: LEADER
I20260619 10:25:35.411736 8470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:35.411760 8475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260619 10:25:35.412128 8475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487f479908bf40629dea94766fce1a22. Latest consensus state: current_term: 1 leader_uuid: "487f479908bf40629dea94766fce1a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } } }
I20260619 10:25:35.412139 8476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "487f479908bf40629dea94766fce1a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487f479908bf40629dea94766fce1a22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } } }
I20260619 10:25:35.412279 8475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:35.412308 8476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22 [sys.catalog]: This master's current role is: LEADER
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260619 10:25:35.414011 8491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:35.414098 8491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:35.414161 8481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:35.414325 8481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:35.415853 8481 catalog_manager.cc:1347] Generated new cluster ID: 00c4298ea3164b6dad5941f80d3d4405
I20260619 10:25:35.415904 8481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:35.416585 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:35.405616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:60351 (local address 127.0.0.1:46397)
0619 10:25:35.405807 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:35.405816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:35.405846 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:25:35.406323 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:35.406329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:35.406376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:35.406521 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:35.407342 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:35.408726 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:35.409479 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:35.410123 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:35.410244 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:35.413020 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:35.413024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:35.413024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:35.413037 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:35.414865 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:35.415361 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:35.415363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:35.415364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:35.415454 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:35.415710 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:35.415711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:35.415712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:35.415968 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:35.416161 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:35.416290 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 10:25:35.416389 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260619 10:25:35.419796 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:35.427070 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:35.421167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:39501 (local address 127.0.0.1:46397)
0619 10:25:35.421207 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:35.421208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:35.421289 (+ 81us) server_negotiation.cc:407] Connection header received
0619 10:25:35.421313 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:35.421314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:35.421323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:35.421366 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:35.421867 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:35.422352 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:35.423068 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:35.423344 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:35.423435 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:35.424549 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:35.424550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:35.424551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:35.424557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:35.425910 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:35.426281 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:35.426282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:35.426282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:35.426333 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:35.426558 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:35.426559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:35.426559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:35.426663 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:35.426773 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:35.426902 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 10:25:35.426961 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:25:35.429687 8481 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:35.430150 8481 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:35.434185 8481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 487f479908bf40629dea94766fce1a22: Generated new TSK 0
I20260619 10:25:35.434370 8481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:35.436244 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8402
W20260619 10:25:35.440011 8387 connection.cc:535] client connection to 127.0.0.1:46397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46397): Connection reset by peer (error 104)
2026-06-19T10:25:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8501](info): set up 2 sockets
Jun 19 10:25:35 dist-test-slave-xd2q krb5kdc[8501](info): commencing operation
krb5kdc: starting...
W20260619 10:25:36.459446 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:36 dist-test-slave-xd2q krb5kdc[8501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:36.568375 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:36.647328 8514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:36.647555 8514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:36.647594 8514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:36.648725 8514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:36.648800 8514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:36.648830 8514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:36.648866 8514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:36.648900 8514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:36.650386 8514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:36.650664 8514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:36.650882 8514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:36.655412 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:36.655467 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:36.655438 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:36.655611 8514 server_base.cc:1034] running on GCE node
I20260619 10:25:36.655867 8514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:36.656144 8514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:36.657323 8514 hybrid_clock.cc:648] HybridClock initialized: now 1781864736657312 us; error 27 us; skew 500 ppm
Jun 19 10:25:36 dist-test-slave-xd2q krb5kdc[8501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:36.659615 8514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:36.660123 8514 webserver.cc:460] Webserver started at http://127.0.0.1:46803/ using document root <none> and password file <none>
I20260619 10:25:36.660346 8514 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:36.660418 8514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:36.660570 8514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:36.661541 8514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "5bb560162bc24ed39c0603dee5fe31e9"
format_stamp: "Formatted at 2026-06-19 10:25:36 on dist-test-slave-xd2q"
server_key: "05bebeb1126c53a0346484b33b7ecadb"
server_key_iv: "4207fa64247d0111ca231784fb2afc6e"
server_key_version: "encryptionkey@0"
I20260619 10:25:36.661893 8514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "5bb560162bc24ed39c0603dee5fe31e9"
format_stamp: "Formatted at 2026-06-19 10:25:36 on dist-test-slave-xd2q"
server_key: "05bebeb1126c53a0346484b33b7ecadb"
server_key_iv: "4207fa64247d0111ca231784fb2afc6e"
server_key_version: "encryptionkey@0"
I20260619 10:25:36.663163 8514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:36.663980 8529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:36.664156 8514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:36.664265 8514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "5bb560162bc24ed39c0603dee5fe31e9"
format_stamp: "Formatted at 2026-06-19 10:25:36 on dist-test-slave-xd2q"
server_key: "05bebeb1126c53a0346484b33b7ecadb"
server_key_iv: "4207fa64247d0111ca231784fb2afc6e"
server_key_version: "encryptionkey@0"
I20260619 10:25:36.664338 8514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:36.681978 8514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:36.683759 8514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:36.683928 8514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:36.687845 8514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43483
I20260619 10:25:36.687891 8581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43483 every 8 connection(s)
I20260619 10:25:36.688220 8514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:36.689009 8582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:36.691090 8582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: Bootstrap starting.
I20260619 10:25:36.691751 8582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:36.692060 8582 log.cc:824] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:36.692130 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8514
I20260619 10:25:36.692267 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:36.692483 19718 external_mini_cluster.cc:1428] Setting key 2f94949b3846798a1e4eae991154e0f1
I20260619 10:25:36.692826 8582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: No bootstrap required, opened a new log
I20260619 10:25:36.694515 8582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260619 10:25:36.694670 8582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:36.694736 8582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb560162bc24ed39c0603dee5fe31e9, State: Initialized, Role: FOLLOWER
I20260619 10:25:36.694876 8582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260619 10:25:36.694989 8582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:36.695029 8582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:36.695081 8582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:36.695690 8582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260619 10:25:36.695817 8582 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb560162bc24ed39c0603dee5fe31e9; no voters:
I20260619 10:25:36.695971 8582 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:36.695992 8587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:36.696095 8587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb560162bc24ed39c0603dee5fe31e9, State: Running, Role: LEADER
I20260619 10:25:36.696218 8582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:36.696225 8587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260619 10:25:36.697865 8588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb560162bc24ed39c0603dee5fe31e9. Latest consensus state: current_term: 1 leader_uuid: "5bb560162bc24ed39c0603dee5fe31e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
I20260619 10:25:36.697999 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb560162bc24ed39c0603dee5fe31e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb560162bc24ed39c0603dee5fe31e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
W20260619 10:25:36.698141 8602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 10:25:36.698591 8588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:36.698639 8602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:36.698599 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:36.699084 8604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:36.699299 8604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:36.699599 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:36.692985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:34733 (local address 127.0.0.1:43483)
0619 10:25:36.693180 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:36.693188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:36.693224 (+ 36us) server_negotiation.cc:407] Connection header received
0619 10:25:36.694505 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:36.694511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:36.694557 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:36.694698 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:36.694921 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:36.696387 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:36.697576 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:36.698171 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:36.698283 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:36.698912 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:36.698915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:36.698916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:36.699103 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:36.699137 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:36.699150 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:36.699233 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:36.699369 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 10:25:36.699396 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260619 10:25:36.700469 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:36.700774 8604 catalog_manager.cc:1347] Generated new cluster ID: 97e37bdf9ac347b5800877fa583a2315
I20260619 10:25:36.700819 8604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:36.705380 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:36.701950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:39551 (local address 127.0.0.1:43483)
0619 10:25:36.701980 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:36.701981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:36.702094 (+ 113us) server_negotiation.cc:407] Connection header received
0619 10:25:36.702144 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:36.702145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:36.702153 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:36.702206 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:36.702339 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:36.703025 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:36.704082 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:36.704406 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:36.704471 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:36.705123 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:36.705124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:36.705125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:36.705128 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:36.705140 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:36.705144 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:36.705197 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:36.705301 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:25:36.705312 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:25:36.712001 8604 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:36.712492 8604 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:36.717780 8604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bb560162bc24ed39c0603dee5fe31e9: Generated new TSK 0
I20260619 10:25:36.717957 8604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:36.726336 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8514
W20260619 10:25:36.730312 8499 connection.cc:535] client connection to 127.0.0.1:43483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43483): Connection reset by peer (error 104)
2026-06-19T10:25:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:36 dist-test-slave-xd2q krb5kdc[8613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:36 dist-test-slave-xd2q krb5kdc[8613](info): set up 2 sockets
Jun 19 10:25:36 dist-test-slave-xd2q krb5kdc[8613](info): commencing operation
krb5kdc: starting...
W20260619 10:25:37.747268 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:37 dist-test-slave-xd2q krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:25:37.858978 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:37.939723 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:37.939942 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:37.939981 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:37.941102 8626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:37.941169 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:37.941198 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:37.941233 8626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:37.941259 8626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:37.942698 8626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:37.942957 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:37.943176 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:37.947824 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:37.947855 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:37.947858 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:37.948004 8626 server_base.cc:1034] running on GCE node
I20260619 10:25:37.948208 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:37.948486 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:37.949645 8626 hybrid_clock.cc:648] HybridClock initialized: now 1781864737949635 us; error 48 us; skew 500 ppm
Jun 19 10:25:37 dist-test-slave-xd2q krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:37.951992 8626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:25:37.952419 8626 webserver.cc:460] Webserver started at http://127.0.0.1:46597/ using document root <none> and password file <none>
I20260619 10:25:37.952627 8626 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:37.952690 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:37.952827 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:37.953790 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "e19f5396cde04c42880bf01aa21dae3d"
format_stamp: "Formatted at 2026-06-19 10:25:37 on dist-test-slave-xd2q"
server_key: "833b5eff98b3d75fadd3bd8ffa2cc5c6"
server_key_iv: "2290f22e8b3bb8a92fdb4a25b1196151"
server_key_version: "encryptionkey@0"
I20260619 10:25:37.954121 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "e19f5396cde04c42880bf01aa21dae3d"
format_stamp: "Formatted at 2026-06-19 10:25:37 on dist-test-slave-xd2q"
server_key: "833b5eff98b3d75fadd3bd8ffa2cc5c6"
server_key_iv: "2290f22e8b3bb8a92fdb4a25b1196151"
server_key_version: "encryptionkey@0"
I20260619 10:25:37.955412 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:37.956113 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:37.956280 8626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:37.956369 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "e19f5396cde04c42880bf01aa21dae3d"
format_stamp: "Formatted at 2026-06-19 10:25:37 on dist-test-slave-xd2q"
server_key: "833b5eff98b3d75fadd3bd8ffa2cc5c6"
server_key_iv: "2290f22e8b3bb8a92fdb4a25b1196151"
server_key_version: "encryptionkey@0"
I20260619 10:25:37.956445 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:37.959631 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:37.961519 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:37.961617 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:37.965328 8626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44959
I20260619 10:25:37.965372 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44959 every 8 connection(s)
I20260619 10:25:37.965655 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:37.966398 8694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:37.968508 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d: Bootstrap starting.
I20260619 10:25:37.969033 8694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:37.969302 8694 log.cc:824] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:37.969929 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d: No bootstrap required, opened a new log
I20260619 10:25:37.971478 8694 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20260619 10:25:37.971618 8694 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:37.971662 8694 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19f5396cde04c42880bf01aa21dae3d, State: Initialized, Role: FOLLOWER
I20260619 10:25:37.971828 8694 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20260619 10:25:37.971917 8694 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:37.971959 8694 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:37.972008 8694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:37.972481 8694 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20260619 10:25:37.972592 8694 leader_election.cc:302] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19f5396cde04c42880bf01aa21dae3d; no voters:
I20260619 10:25:37.972754 8694 leader_election.cc:288] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:37.972766 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8626
I20260619 10:25:37.972797 8697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:37.972877 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:37.972990 8694 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:37.973001 8697 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [term 1 LEADER]: Becoming Leader. State: Replica: e19f5396cde04c42880bf01aa21dae3d, State: Running, Role: LEADER
I20260619 10:25:37.973070 19718 external_mini_cluster.cc:1428] Setting key a91174d5b299fd7587f997a5d006efec
I20260619 10:25:37.973130 8697 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20260619 10:25:37.973644 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19f5396cde04c42880bf01aa21dae3d. Latest consensus state: current_term: 1 leader_uuid: "e19f5396cde04c42880bf01aa21dae3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } } }
I20260619 10:25:37.973728 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:37.973759 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19f5396cde04c42880bf01aa21dae3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19f5396cde04c42880bf01aa21dae3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } } }
I20260619 10:25:37.973843 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:37.974462 8706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:37.974960 8706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:37.975481 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:37.973481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:48753 (local address 127.0.0.1:44959)
0619 10:25:37.973785 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:37.973796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:37.973832 (+ 36us) server_negotiation.cc:407] Connection header received
0619 10:25:37.973942 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:37.973948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:37.973978 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:37.974114 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:37.974798 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:37.974802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:37.974802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:37.974808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:37.974857 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:37.974874 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:37.975247 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:37.975282 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 10:25:37.975346 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":71,"threads_started":1}
I20260619 10:25:37.976411 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:37.979671 8706 catalog_manager.cc:1347] Generated new cluster ID: a44d8a6af4c14be79df5e44a06ef91b8
I20260619 10:25:37.979740 8706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:37.980716 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:37.978097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:33695 (local address 127.0.0.1:44959)
0619 10:25:37.978135 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:37.978137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:37.979785 (+ 1648us) server_negotiation.cc:407] Connection header received
0619 10:25:37.979870 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:37.979872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:37.979877 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:37.979950 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:37.980493 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:37.980494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:37.980494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:37.980498 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:37.980512 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:37.980517 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:37.980573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:37.980632 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 10:25:37.980645 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:25:37.989149 8706 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:37.989328 8706 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:37.994769 8706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19f5396cde04c42880bf01aa21dae3d: Generated new TSK 0
I20260619 10:25:37.994927 8706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:37.996460 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8626
2026-06-19T10:25:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:38 dist-test-slave-xd2q krb5kdc[8725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:38 dist-test-slave-xd2q krb5kdc[8725](info): set up 2 sockets
Jun 19 10:25:38 dist-test-slave-xd2q krb5kdc[8725](info): commencing operation
krb5kdc: starting...
W20260619 10:25:39.003513 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:39.112439 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33333
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:39.187557 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:39.187788 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:39.187824 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:39.188871 8738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:39.188935 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:39.188958 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:39.188982 8738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:39.189009 8738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:39.190318 8738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:33333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:39.190587 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:39.190785 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:39.195281 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:39.195271 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:39.195366 8738 server_base.cc:1034] running on GCE node
W20260619 10:25:39.195271 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:39.195791 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:39.196123 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:39.197304 8738 hybrid_clock.cc:648] HybridClock initialized: now 1781864739197289 us; error 25 us; skew 500 ppm
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864739, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:39.199710 8738 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:39.200137 8738 webserver.cc:460] Webserver started at http://127.19.65.190:43553/ using document root <none> and password file <none>
I20260619 10:25:39.200348 8738 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:39.200413 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:39.200541 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:39.201507 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "a971b0bd1d4045b4aeea1c1af4fde268"
format_stamp: "Formatted at 2026-06-19 10:25:39 on dist-test-slave-xd2q"
server_key: "ea564b74b2ba5fbcc5dc526568a38723"
server_key_iv: "743837f8897b326fd059d6e68ddb368b"
server_key_version: "encryptionkey@0"
I20260619 10:25:39.201893 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "a971b0bd1d4045b4aeea1c1af4fde268"
format_stamp: "Formatted at 2026-06-19 10:25:39 on dist-test-slave-xd2q"
server_key: "ea564b74b2ba5fbcc5dc526568a38723"
server_key_iv: "743837f8897b326fd059d6e68ddb368b"
server_key_version: "encryptionkey@0"
I20260619 10:25:39.203143 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:39.203971 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:39.204160 8738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:25:39.204263 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "a971b0bd1d4045b4aeea1c1af4fde268"
format_stamp: "Formatted at 2026-06-19 10:25:39 on dist-test-slave-xd2q"
server_key: "ea564b74b2ba5fbcc5dc526568a38723"
server_key_iv: "743837f8897b326fd059d6e68ddb368b"
server_key_version: "encryptionkey@0"
I20260619 10:25:39.204355 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:39.221364 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:39.223052 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:39.223207 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:39.227046 8738 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:33333
I20260619 10:25:39.227087 8805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:33333 every 8 connection(s)
I20260619 10:25:39.227487 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:39.228324 8806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:39.230418 8806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: Bootstrap starting.
I20260619 10:25:39.230973 8806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:39.231277 8806 log.cc:824] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:39.231994 8806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: No bootstrap required, opened a new log
I20260619 10:25:39.233568 8806 raft_consensus.cc:348] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } }
I20260619 10:25:39.233731 8806 raft_consensus.cc:374] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:39.233776 8806 raft_consensus.cc:729] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a971b0bd1d4045b4aeea1c1af4fde268, State: Initialized, Role: FOLLOWER
I20260619 10:25:39.233922 8806 consensus_queue.cc:260] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } }
I20260619 10:25:39.234004 8806 raft_consensus.cc:388] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:39.234042 8806 raft_consensus.cc:482] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:39.234093 8806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:39.234895 8806 raft_consensus.cc:504] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } }
I20260619 10:25:39.235009 8806 leader_election.cc:302] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a971b0bd1d4045b4aeea1c1af4fde268; no voters:
I20260619 10:25:39.235188 8806 leader_election.cc:288] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:39.235244 8809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:39.235445 8806 sys_catalog.cc:564] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:39.235452 8809 raft_consensus.cc:686] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [term 1 LEADER]: Becoming Leader. State: Replica: a971b0bd1d4045b4aeea1c1af4fde268, State: Running, Role: LEADER
I20260619 10:25:39.235621 8809 consensus_queue.cc:237] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } }
I20260619 10:25:39.235987 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } } }
I20260619 10:25:39.236088 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:39.236194 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a971b0bd1d4045b4aeea1c1af4fde268. Latest consensus state: current_term: 1 leader_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971b0bd1d4045b4aeea1c1af4fde268" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33333 } } }
I20260619 10:25:39.236336 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:39.236675 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8738
I20260619 10:25:39.236790 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:39.236975 19718 external_mini_cluster.cc:1428] Setting key c07c615e98907596eff6784f4289ad09
W20260619 10:25:39.237215 8825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:39.237278 8825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:39.237700 8818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:39.237877 8818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:39.239238 8818 catalog_manager.cc:1347] Generated new cluster ID: 4114ac77eb17481d813bae1fc105a4e5
I20260619 10:25:39.239312 8818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:39.248052 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:39.237608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.19.65.190:33333)
0619 10:25:39.239431 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:39.239437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:39.239460 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:25:39.239548 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:39.239553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:39.239590 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:39.239728 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:39.240371 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:39.241275 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:39.242022 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:39.242459 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:39.242571 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:39.244915 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:39.244918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:39.244918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:39.244936 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:39.246525 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:39.246913 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:39.246916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:39.246916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:39.246998 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:39.247227 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:39.247228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:39.247228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:39.247493 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:39.247696 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:39.247778 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 10:25:39.247876 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1770,"thread_start_us":76,"threads_started":1}
I20260619 10:25:39.248960 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:39.253700 8818 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:39.254346 8818 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:39.259912 8818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a971b0bd1d4045b4aeea1c1af4fde268: Generated new TSK 0
I20260619 10:25:39.260064 8818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:39.260921 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:39.253763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54641 (local address 127.19.65.190:33333)
0619 10:25:39.253799 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:39.253801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:39.253937 (+ 136us) server_negotiation.cc:407] Connection header received
0619 10:25:39.253983 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:39.253984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:39.253994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:39.254055 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:39.254672 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:39.255384 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:39.256407 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:39.256708 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:39.256767 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:39.258108 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:39.258109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:39.258109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:39.258115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:39.259733 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:39.260146 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:39.260147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:39.260147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:39.260214 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:39.260449 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:39.260449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:39.260450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:39.260572 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:39.260673 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:39.260777 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:25:39.260840 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:39.262722 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8738
W20260619 10:25:39.266781 8723 connection.cc:535] client connection to 127.19.65.190:33333 recv error: Network error: failed to read from TLS socket (remote: 127.19.65.190:33333): Connection reset by peer (error 104)
2026-06-19T10:25:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8837](info): set up 2 sockets
Jun 19 10:25:39 dist-test-slave-xd2q krb5kdc[8837](info): commencing operation
krb5kdc: starting...
W20260619 10:25:40.279418 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:40 dist-test-slave-xd2q krb5kdc[8837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:40.385984 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32927
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:40.461778 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:40.462008 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:40.462042 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:40.463121 8850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:40.463186 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:40.463212 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:40.463240 8850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:40.463263 8850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:40.464686 8850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:40.464974 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:40.465184 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:40.469791 8858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:40.469803 8856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:40.469831 8850 server_base.cc:1034] running on GCE node
W20260619 10:25:40.469803 8855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:40.470295 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:40.470597 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:40.471746 8850 hybrid_clock.cc:648] HybridClock initialized: now 1781864740471742 us; error 35 us; skew 500 ppm
Jun 19 10:25:40 dist-test-slave-xd2q krb5kdc[8837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864740, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:40.477653 8850 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:40.478029 8850 webserver.cc:460] Webserver started at http://127.19.65.190:40025/ using document root <none> and password file <none>
I20260619 10:25:40.478257 8850 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:40.478324 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:40.478503 8850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:40.479538 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "51b5f3cfac334078be37d82ac0a69d11"
format_stamp: "Formatted at 2026-06-19 10:25:40 on dist-test-slave-xd2q"
server_key: "045ae5bc8dab3ee25586c2bf3b77dac4"
server_key_iv: "343be22b419959eef4ae39b38eaccbe3"
server_key_version: "encryptionkey@0"
I20260619 10:25:40.479909 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "51b5f3cfac334078be37d82ac0a69d11"
format_stamp: "Formatted at 2026-06-19 10:25:40 on dist-test-slave-xd2q"
server_key: "045ae5bc8dab3ee25586c2bf3b77dac4"
server_key_iv: "343be22b419959eef4ae39b38eaccbe3"
server_key_version: "encryptionkey@0"
I20260619 10:25:40.481105 8850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:40.481757 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:40.481933 8850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:40.482000 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "51b5f3cfac334078be37d82ac0a69d11"
format_stamp: "Formatted at 2026-06-19 10:25:40 on dist-test-slave-xd2q"
server_key: "045ae5bc8dab3ee25586c2bf3b77dac4"
server_key_iv: "343be22b419959eef4ae39b38eaccbe3"
server_key_version: "encryptionkey@0"
I20260619 10:25:40.482084 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:40.494936 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:40.497056 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:40.497208 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:40.500895 8850 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:32927
I20260619 10:25:40.500923 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:32927 every 8 connection(s)
I20260619 10:25:40.501230 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:40.502022 8918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:40.504179 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11: Bootstrap starting.
I20260619 10:25:40.504739 8918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:40.505021 8918 log.cc:824] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:40.505684 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11: No bootstrap required, opened a new log
I20260619 10:25:40.507172 8918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } }
I20260619 10:25:40.507346 8918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:40.507401 8918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b5f3cfac334078be37d82ac0a69d11, State: Initialized, Role: FOLLOWER
I20260619 10:25:40.507555 8918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } }
I20260619 10:25:40.507635 8918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:40.507690 8918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:40.507740 8918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:40.508493 8918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } }
I20260619 10:25:40.508600 8918 leader_election.cc:302] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b5f3cfac334078be37d82ac0a69d11; no voters:
I20260619 10:25:40.508787 8918 leader_election.cc:288] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:40.508808 8921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:40.508912 8921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [term 1 LEADER]: Becoming Leader. State: Replica: 51b5f3cfac334078be37d82ac0a69d11, State: Running, Role: LEADER
I20260619 10:25:40.508989 8921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } }
I20260619 10:25:40.509024 8918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:40.509392 8922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b5f3cfac334078be37d82ac0a69d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } } }
I20260619 10:25:40.509408 8923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b5f3cfac334078be37d82ac0a69d11. Latest consensus state: current_term: 1 leader_uuid: "51b5f3cfac334078be37d82ac0a69d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5f3cfac334078be37d82ac0a69d11" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 32927 } } }
I20260619 10:25:40.509580 8923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:40.509517 8922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:40.509943 8928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:40.510208 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8850
I20260619 10:25:40.510324 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:40.510447 8928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:40.510507 19718 external_mini_cluster.cc:1428] Setting key 2e70cf96a78114c87face895115df0ee
I20260619 10:25:40.512061 8928 catalog_manager.cc:1347] Generated new cluster ID: a057abb9a3ae42e093b72045aa32f550
I20260619 10:25:40.512115 8928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:40.515609 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:40.510883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32847 (local address 127.19.65.190:32927)
0619 10:25:40.511316 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:40.511325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:40.511356 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:25:40.511458 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:40.511464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:40.511521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:40.511648 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:40.511869 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:40.512839 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:40.513633 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:40.514047 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:40.514162 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:40.514839 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:40.514843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:40.514843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:40.515013 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:40.515041 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:40.515051 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:40.515142 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:40.515243 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:25:40.515270 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":79,"threads_started":1}
I20260619 10:25:40.516623 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:40.522248 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:40.518071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41739 (local address 127.19.65.190:32927)
0619 10:25:40.518108 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:40.518110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:40.519540 (+ 1430us) server_negotiation.cc:407] Connection header received
0619 10:25:40.519609 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:40.519610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:40.519620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:40.519672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:40.519832 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:40.520308 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:40.521013 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:40.521252 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:40.521342 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:40.521981 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:40.521982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:40.521982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:40.521986 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:40.521998 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:40.522002 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:40.522057 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:40.522127 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:40.522139 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:25:40.533746 8928 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:40.534201 8928 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:40.538748 8928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b5f3cfac334078be37d82ac0a69d11: Generated new TSK 0
I20260619 10:25:40.538928 8928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:40.549111 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8850
W20260619 10:25:40.553066 8835 connection.cc:535] client connection to 127.19.65.190:32927 recv error: Network error: failed to read from TLS socket (remote: 127.19.65.190:32927): Connection reset by peer (error 104)
2026-06-19T10:25:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:40 dist-test-slave-xd2q krb5kdc[8949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:40 dist-test-slave-xd2q krb5kdc[8949](info): set up 2 sockets
Jun 19 10:25:40 dist-test-slave-xd2q krb5kdc[8949](info): commencing operation
krb5kdc: starting...
W20260619 10:25:41.559381 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:41 dist-test-slave-xd2q krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:41.674465 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36799
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:36799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:41.753217 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:41.753407 8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:41.753453 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:41.754554 8962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:41.754593 8962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:41.754606 8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:41.754619 8962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:41.754688 8962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:41.756178 8962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:36799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:36799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:41.756392 8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:41.756593 8962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:41.761204 8967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:41.761309 8970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:41.761214 8968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:41.761219 8962 server_base.cc:1034] running on GCE node
I20260619 10:25:41.761686 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:41.761968 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:41.763120 8962 hybrid_clock.cc:648] HybridClock initialized: now 1781864741763107 us; error 33 us; skew 500 ppm
Jun 19 10:25:41 dist-test-slave-xd2q krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864741, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:41.765537 8962 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:41.765978 8962 webserver.cc:460] Webserver started at http://127.19.65.190:32951/ using document root <none> and password file <none>
I20260619 10:25:41.766201 8962 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:41.766268 8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:41.766394 8962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:41.767449 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "1a83c2467e3048cfbaec0b5b32ef9444"
format_stamp: "Formatted at 2026-06-19 10:25:41 on dist-test-slave-xd2q"
server_key: "46e7481e87fa3883c360159d9d90213b"
server_key_iv: "6038879bd8974f32959405f7dc06b3d4"
server_key_version: "encryptionkey@0"
I20260619 10:25:41.767814 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "1a83c2467e3048cfbaec0b5b32ef9444"
format_stamp: "Formatted at 2026-06-19 10:25:41 on dist-test-slave-xd2q"
server_key: "46e7481e87fa3883c360159d9d90213b"
server_key_iv: "6038879bd8974f32959405f7dc06b3d4"
server_key_version: "encryptionkey@0"
I20260619 10:25:41.769058 8962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:41.769711 8977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:41.769877 8962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:41.769977 8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "1a83c2467e3048cfbaec0b5b32ef9444"
format_stamp: "Formatted at 2026-06-19 10:25:41 on dist-test-slave-xd2q"
server_key: "46e7481e87fa3883c360159d9d90213b"
server_key_iv: "6038879bd8974f32959405f7dc06b3d4"
server_key_version: "encryptionkey@0"
I20260619 10:25:41.770030 8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:41.772954 8962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:41.774797 8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:41.774897 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:41.778340 8962 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:36799
I20260619 10:25:41.778386 9029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:36799 every 8 connection(s)
I20260619 10:25:41.778646 8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:41.778898 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 8962
I20260619 10:25:41.779004 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:41.779192 19718 external_mini_cluster.cc:1428] Setting key 6ccd6234add012a9e94a3fb7b7ba0b11
I20260619 10:25:41.779378 9030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:41.781548 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:41.779712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39001 (local address 127.19.65.190:36799)
0619 10:25:41.779884 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:41.779890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:41.779960 (+ 70us) server_negotiation.cc:407] Connection header received
0619 10:25:41.780442 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:41.780447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:41.780469 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:41.780569 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:41.781129 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:41.781132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:41.781132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:41.781135 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:41.781166 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:41.781177 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:41.781228 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:41.781317 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 10:25:41.781363 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260619 10:25:41.782400 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:41.782593 9030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: Bootstrap starting.
I20260619 10:25:41.783298 9030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:41.783670 9030 log.cc:824] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:41.784402 9030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: No bootstrap required, opened a new log
I20260619 10:25:41.786054 9030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } }
I20260619 10:25:41.786176 9030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:41.786197 9030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a83c2467e3048cfbaec0b5b32ef9444, State: Initialized, Role: FOLLOWER
I20260619 10:25:41.786315 9030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } }
I20260619 10:25:41.786399 9030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:41.786422 9030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:41.786448 9030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:41.786906 9030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } }
I20260619 10:25:41.786985 9030 leader_election.cc:302] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a83c2467e3048cfbaec0b5b32ef9444; no voters:
I20260619 10:25:41.787094 9030 leader_election.cc:288] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:41.787149 9040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:41.787329 9040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [term 1 LEADER]: Becoming Leader. State: Replica: 1a83c2467e3048cfbaec0b5b32ef9444, State: Running, Role: LEADER
I20260619 10:25:41.787385 9030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:41.787452 9040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } }
I20260619 10:25:41.787806 9041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } } }
I20260619 10:25:41.787806 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a83c2467e3048cfbaec0b5b32ef9444. Latest consensus state: current_term: 1 leader_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a83c2467e3048cfbaec0b5b32ef9444" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 36799 } } }
I20260619 10:25:41.787945 9041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:41.788007 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444 [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:41.789187 9056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:41.789242 9056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:41.789296 9049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:41.789498 9049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:41.789726 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:41.783999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45075 (local address 127.19.65.190:36799)
0619 10:25:41.784042 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:41.784044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:41.784143 (+ 99us) server_negotiation.cc:407] Connection header received
0619 10:25:41.788790 (+ 4647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:41.788792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:41.788797 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:41.788851 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:41.789531 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:41.789532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:41.789532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:41.789536 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:41.789547 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:41.789550 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:41.789618 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:41.789678 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 10:25:41.789688 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:25:41.790915 9049 catalog_manager.cc:1347] Generated new cluster ID: b965738cc6aa49198c54a00299d21240
I20260619 10:25:41.790975 9049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:41.796554 9049 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:41.796708 9049 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:41.799626 9049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a83c2467e3048cfbaec0b5b32ef9444: Generated new TSK 0
I20260619 10:25:41.799782 9049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:41.801951 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 8962
2026-06-19T10:25:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:41 dist-test-slave-xd2q krb5kdc[9061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:41 dist-test-slave-xd2q krb5kdc[9061](info): set up 2 sockets
Jun 19 10:25:41 dist-test-slave-xd2q krb5kdc[9061](info): commencing operation
krb5kdc: starting...
W20260619 10:25:42.815523 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:42 dist-test-slave-xd2q krb5kdc[9061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:42.929488 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39483
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:39483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:43.008687 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:43.008867 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:43.008888 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:43.009980 9074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:43.010017 9074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:43.010030 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:43.010066 9074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:43.010092 9074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:43.011449 9074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:39483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:43.011663 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:43.011858 9074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:43.016472 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:43.016484 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:43.016527 9074 server_base.cc:1034] running on GCE node
W20260619 10:25:43.016484 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:43.016930 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:43.017167 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:43.018335 9074 hybrid_clock.cc:648] HybridClock initialized: now 1781864743018306 us; error 39 us; skew 500 ppm
Jun 19 10:25:43 dist-test-slave-xd2q krb5kdc[9061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:43.020615 9074 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:43.021005 9074 webserver.cc:460] Webserver started at http://127.19.65.190:41807/ using document root <none> and password file <none>
I20260619 10:25:43.021267 9074 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:43.021337 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:43.021488 9074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:43.022517 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "84bb0697977043b28ad77fe0e3a473e9"
format_stamp: "Formatted at 2026-06-19 10:25:43 on dist-test-slave-xd2q"
server_key: "a5540262ba18701f0cb9aa9bb499fea8"
server_key_iv: "084a2062adec331f3b51d5c68fc15a73"
server_key_version: "encryptionkey@0"
I20260619 10:25:43.022874 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "84bb0697977043b28ad77fe0e3a473e9"
format_stamp: "Formatted at 2026-06-19 10:25:43 on dist-test-slave-xd2q"
server_key: "a5540262ba18701f0cb9aa9bb499fea8"
server_key_iv: "084a2062adec331f3b51d5c68fc15a73"
server_key_version: "encryptionkey@0"
I20260619 10:25:43.024366 9074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:43.025072 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:43.025241 9074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:43.025332 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "84bb0697977043b28ad77fe0e3a473e9"
format_stamp: "Formatted at 2026-06-19 10:25:43 on dist-test-slave-xd2q"
server_key: "a5540262ba18701f0cb9aa9bb499fea8"
server_key_iv: "084a2062adec331f3b51d5c68fc15a73"
server_key_version: "encryptionkey@0"
I20260619 10:25:43.025417 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:43.047857 9074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:43.049691 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:43.049852 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:43.054065 9074 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:39483
I20260619 10:25:43.054097 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:39483 every 8 connection(s)
I20260619 10:25:43.054436 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:43.055262 9142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:43.057308 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9: Bootstrap starting.
I20260619 10:25:43.057801 9142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:43.058038 9142 log.cc:824] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:43.058717 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9: No bootstrap required, opened a new log
I20260619 10:25:43.060299 9142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } }
I20260619 10:25:43.060420 9142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:43.060441 9142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bb0697977043b28ad77fe0e3a473e9, State: Initialized, Role: FOLLOWER
I20260619 10:25:43.060557 9142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } }
I20260619 10:25:43.060635 9142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:43.060657 9142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:43.060681 9142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:43.061375 9142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } }
I20260619 10:25:43.061451 9142 leader_election.cc:302] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84bb0697977043b28ad77fe0e3a473e9; no voters:
I20260619 10:25:43.061564 9142 leader_election.cc:288] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:43.061656 9145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:43.061774 9142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:43.061821 9145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [term 1 LEADER]: Becoming Leader. State: Replica: 84bb0697977043b28ad77fe0e3a473e9, State: Running, Role: LEADER
I20260619 10:25:43.062011 9145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } }
I20260619 10:25:43.062372 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84bb0697977043b28ad77fe0e3a473e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } } }
I20260619 10:25:43.062371 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84bb0697977043b28ad77fe0e3a473e9. Latest consensus state: current_term: 1 leader_uuid: "84bb0697977043b28ad77fe0e3a473e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bb0697977043b28ad77fe0e3a473e9" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 39483 } } }
I20260619 10:25:43.062570 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:43.062577 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:43.062985 9154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:43.063479 9154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:43.063836 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9074
I20260619 10:25:43.063953 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:43.064169 19718 external_mini_cluster.cc:1428] Setting key 8f7e284890325a35269380b19eb3d482
I20260619 10:25:43.065114 9154 catalog_manager.cc:1347] Generated new cluster ID: c25f32a8611649009efb54d2d717465a
I20260619 10:25:43.065172 9154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:25:43 dist-test-slave-xd2q krb5kdc[9061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:43.073505 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:43.064620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37255 (local address 127.19.65.190:39483)
0619 10:25:43.064942 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:43.064950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:43.064980 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:25:43.065103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:43.065120 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:43.065175 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:43.065304 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:43.065919 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:43.066785 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:43.067582 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:43.067999 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:43.068128 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:43.070411 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:43.070415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:43.070415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:43.070426 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:43.072045 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:43.072390 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:43.072393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:43.072393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:43.072488 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:43.072691 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:43.072692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:43.072692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:43.072903 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:43.073073 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:43.073137 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 10:25:43.073268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":82,"threads_started":1}
I20260619 10:25:43.074321 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:43.081398 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:43.075600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43203 (local address 127.19.65.190:39483)
0619 10:25:43.075662 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:43.075664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:43.075758 (+ 94us) server_negotiation.cc:407] Connection header received
0619 10:25:43.075814 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:43.075815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:43.075825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:43.075878 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:43.076368 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:43.076886 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:43.077581 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:43.077790 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:43.077859 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:43.078999 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:43.079000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:43.079001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:43.079007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:43.080364 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:43.080705 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:43.080706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:43.080706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:43.080754 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:43.080952 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:43.080953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:43.080953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:43.081052 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:43.081154 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:43.081242 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 10:25:43.081308 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:25:43.091063 9154 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:43.091616 9154 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:43.094908 9154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84bb0697977043b28ad77fe0e3a473e9: Generated new TSK 0
I20260619 10:25:43.095085 9154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:43.102640 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9074
2026-06-19T10:25:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:43 dist-test-slave-xd2q krb5kdc[9173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:43 dist-test-slave-xd2q krb5kdc[9173](info): set up 2 sockets
Jun 19 10:25:43 dist-test-slave-xd2q krb5kdc[9173](info): commencing operation
krb5kdc: starting...
W20260619 10:25:44.123052 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:44 dist-test-slave-xd2q krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:44.234241 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34371
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:44.313994 9186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:44.314186 9186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:44.314208 9186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:44.315397 9186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:44.315436 9186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:44.315451 9186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:44.315487 9186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:44.315514 9186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:44.317009 9186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:44.317263 9186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:44.317472 9186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:44.322132 9192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:44.322127 9194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:44.322243 9186 server_base.cc:1034] running on GCE node
W20260619 10:25:44.322350 9191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:44.322564 9186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:44.322839 9186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:44.324003 9186 hybrid_clock.cc:648] HybridClock initialized: now 1781864744323984 us; error 41 us; skew 500 ppm
Jun 19 10:25:44 dist-test-slave-xd2q krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864744, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:44.326337 9186 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:44.326776 9186 webserver.cc:460] Webserver started at http://127.19.65.190:33993/ using document root <none> and password file <none>
I20260619 10:25:44.326979 9186 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:44.327042 9186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:44.327183 9186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:44.328198 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "7dfb52740bd640da8203474b6f20f159"
format_stamp: "Formatted at 2026-06-19 10:25:44 on dist-test-slave-xd2q"
server_key: "5295e4e0f99f25a26abd622d4631b858"
server_key_iv: "c95164cfc4552891768e53bb791146d3"
server_key_version: "encryptionkey@0"
I20260619 10:25:44.328552 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "7dfb52740bd640da8203474b6f20f159"
format_stamp: "Formatted at 2026-06-19 10:25:44 on dist-test-slave-xd2q"
server_key: "5295e4e0f99f25a26abd622d4631b858"
server_key_iv: "c95164cfc4552891768e53bb791146d3"
server_key_version: "encryptionkey@0"
I20260619 10:25:44.329772 9186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:25:44.330583 9201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:44.330753 9186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:44.330849 9186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "7dfb52740bd640da8203474b6f20f159"
format_stamp: "Formatted at 2026-06-19 10:25:44 on dist-test-slave-xd2q"
server_key: "5295e4e0f99f25a26abd622d4631b858"
server_key_iv: "c95164cfc4552891768e53bb791146d3"
server_key_version: "encryptionkey@0"
I20260619 10:25:44.330932 9186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:44.351758 9186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:44.353329 9186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:44.353530 9186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:44.357287 9186 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:34371
I20260619 10:25:44.357331 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:34371 every 8 connection(s)
I20260619 10:25:44.357654 9186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:44.358444 9254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:44.358682 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9186
I20260619 10:25:44.358803 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:44.358985 19718 external_mini_cluster.cc:1428] Setting key 78bfcecad3b50f88409748076c1b9272
I20260619 10:25:44.361133 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159: Bootstrap starting.
I20260619 10:25:44.361749 9254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:44.362044 9254 log.cc:824] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:44.362695 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159: No bootstrap required, opened a new log
I20260619 10:25:44.364266 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:44.359512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35371 (local address 127.19.65.190:34371)
0619 10:25:44.359706 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:44.359726 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:44.359749 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:25:44.360153 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:44.360158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:44.360203 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:44.360317 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:44.360587 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:44.361505 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:44.362500 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:44.362918 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:44.362995 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:44.363653 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:44.363657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:44.363657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:44.363813 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:44.363839 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:44.363849 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:44.363898 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:44.364002 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:25:44.364048 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260619 10:25:44.364959 9254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } }
I20260619 10:25:44.365118 9254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:44.365191 9254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfb52740bd640da8203474b6f20f159, State: Initialized, Role: FOLLOWER
I20260619 10:25:44.365304 9254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } }
I20260619 10:25:44.365381 9254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:44.365397 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:44.365423 9254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:44.365473 9254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:44.365950 9254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } }
I20260619 10:25:44.366029 9254 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfb52740bd640da8203474b6f20f159; no voters:
I20260619 10:25:44.366227 9254 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:44.366257 9259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:44.366367 9259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [term 1 LEADER]: Becoming Leader. State: Replica: 7dfb52740bd640da8203474b6f20f159, State: Running, Role: LEADER
I20260619 10:25:44.366441 9259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } }
I20260619 10:25:44.366475 9254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:44.366747 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfb52740bd640da8203474b6f20f159. Latest consensus state: current_term: 1 leader_uuid: "7dfb52740bd640da8203474b6f20f159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } } }
I20260619 10:25:44.366815 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:44.366850 9262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dfb52740bd640da8203474b6f20f159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfb52740bd640da8203474b6f20f159" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 34371 } } }
I20260619 10:25:44.366938 9262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:44.367056 9269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:44.367560 9269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:44.369141 9269 catalog_manager.cc:1347] Generated new cluster ID: 5dc916c6745f4041b618bc0aade60a35
I20260619 10:25:44.369185 9269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:44.370334 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:44.366758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35567 (local address 127.19.65.190:34371)
0619 10:25:44.366793 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:44.366795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:44.367196 (+ 401us) server_negotiation.cc:407] Connection header received
0619 10:25:44.367267 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:44.367268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:44.367278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:44.367348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:44.367575 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:44.368223 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:44.369218 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:44.369422 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:44.369494 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:44.370099 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:44.370100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:44.370100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:44.370104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:44.370116 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:44.370120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:44.370164 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:44.370223 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 10:25:44.370244 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:44.386719 9269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:44.387181 9269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:44.396056 9269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfb52740bd640da8203474b6f20f159: Generated new TSK 0
I20260619 10:25:44.396226 9269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:44.400549 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9186
2026-06-19T10:25:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:44 dist-test-slave-xd2q krb5kdc[9285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:44 dist-test-slave-xd2q krb5kdc[9285](info): set up 2 sockets
Jun 19 10:25:44 dist-test-slave-xd2q krb5kdc[9285](info): commencing operation
krb5kdc: starting...
W20260619 10:25:45.435464 19718 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:45 dist-test-slave-xd2q krb5kdc[9285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:45.545871 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46121
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:45.622578 9298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:45.622797 9298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:45.622833 9298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:45.623967 9298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:45.624037 9298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:45.624065 9298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:45.624095 9298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:45.624118 9298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:45.625519 9298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:45.625777 9298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:45.625984 9298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:45.630610 9303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:45.630604 9306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:45.630585 9304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:45.630705 9298 server_base.cc:1034] running on GCE node
I20260619 10:25:45.631040 9298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:45.631356 9298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:45.632511 9298 hybrid_clock.cc:648] HybridClock initialized: now 1781864745632496 us; error 34 us; skew 500 ppm
Jun 19 10:25:45 dist-test-slave-xd2q krb5kdc[9285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864745, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:45.638597 9298 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:45.639011 9298 webserver.cc:460] Webserver started at http://127.19.65.190:43785/ using document root <none> and password file <none>
I20260619 10:25:45.639181 9298 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:45.639256 9298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:45.639412 9298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:45.640369 9298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "1aa8c9a791e144a59a20633e1c88c8e3"
format_stamp: "Formatted at 2026-06-19 10:25:45 on dist-test-slave-xd2q"
server_key: "5ac3253835102c6e17842d40bedd7885"
server_key_iv: "99975089965d76dac01bd2fc650db14b"
server_key_version: "encryptionkey@0"
I20260619 10:25:45.640661 9298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "1aa8c9a791e144a59a20633e1c88c8e3"
format_stamp: "Formatted at 2026-06-19 10:25:45 on dist-test-slave-xd2q"
server_key: "5ac3253835102c6e17842d40bedd7885"
server_key_iv: "99975089965d76dac01bd2fc650db14b"
server_key_version: "encryptionkey@0"
I20260619 10:25:45.641840 9298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:45.642474 9313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:45.642623 9298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:45.642705 9298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "1aa8c9a791e144a59a20633e1c88c8e3"
format_stamp: "Formatted at 2026-06-19 10:25:45 on dist-test-slave-xd2q"
server_key: "5ac3253835102c6e17842d40bedd7885"
server_key_iv: "99975089965d76dac01bd2fc650db14b"
server_key_version: "encryptionkey@0"
I20260619 10:25:45.642757 9298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:45.645821 9298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:45.647606 9298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:45.647711 9298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:45.651259 9298 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46121
I20260619 10:25:45.651325 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46121 every 8 connection(s)
I20260619 10:25:45.651623 9298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:45.652549 9366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:45.654595 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: Bootstrap starting.
I20260619 10:25:45.655125 9366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:45.655464 9366 log.cc:824] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:45.656091 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: No bootstrap required, opened a new log
I20260619 10:25:45.657605 9366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } }
I20260619 10:25:45.657747 9366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:45.657804 9366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa8c9a791e144a59a20633e1c88c8e3, State: Initialized, Role: FOLLOWER
I20260619 10:25:45.657955 9366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } }
I20260619 10:25:45.658030 9366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:45.658071 9366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:45.658118 9366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:45.658849 9366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } }
I20260619 10:25:45.658977 9366 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa8c9a791e144a59a20633e1c88c8e3; no voters:
I20260619 10:25:45.659147 9366 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:45.659197 9369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:45.659385 9369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1aa8c9a791e144a59a20633e1c88c8e3, State: Running, Role: LEADER
I20260619 10:25:45.659425 9366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:45.659543 9369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } }
I20260619 10:25:45.659592 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9298
I20260619 10:25:45.659703 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:45.659883 19718 external_mini_cluster.cc:1428] Setting key 70e90f121f3a06443dae076a94f752af
I20260619 10:25:45.660053 9371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa8c9a791e144a59a20633e1c88c8e3. Latest consensus state: current_term: 1 leader_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } } }
I20260619 10:25:45.660132 9371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:45.660557 9370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa8c9a791e144a59a20633e1c88c8e3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46121 } } }
I20260619 10:25:45.660640 9370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3 [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:45.661262 9387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:45.661386 9387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:45.661473 9378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:45.661966 9378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:45.662377 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:45.660482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46097 (local address 127.19.65.190:46121)
0619 10:25:45.661037 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:45.661046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:45.661078 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:45.661182 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:45.661187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:45.661216 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:45.661345 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:45.661943 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:45.661946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:45.661947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:45.661951 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:45.662003 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:45.662032 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:45.662121 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:45.662191 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:45.662230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":62,"threads_started":1}
I20260619 10:25:45.663322 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:45.663590 9378 catalog_manager.cc:1347] Generated new cluster ID: 4ae8b4f3de164073b33b6e524f19a7dd
I20260619 10:25:45.663642 9378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:45.665768 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:45.664695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58349 (local address 127.19.65.190:46121)
0619 10:25:45.664739 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:45.664741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:45.664861 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:25:45.664930 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:45.664931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:45.664936 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:45.664991 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:45.665558 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:45.665559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:45.665560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:25:45.665564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:45.665577 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:25:45.665580 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:45.665631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:45.665682 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 10:25:45.665695 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:25:45.683584 9378 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:45.683748 9378 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:45.685626 9378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aa8c9a791e144a59a20633e1c88c8e3: Generated new TSK 0
I20260619 10:25:45.685781 9378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:45.689368 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9298
2026-06-19T10:25:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1288 ms)
[----------] 18 tests from AuthTokenIssuingTest (23340 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:45 dist-test-slave-xd2q krb5kdc[9397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:45 dist-test-slave-xd2q krb5kdc[9397](info): set up 2 sockets
Jun 19 10:25:45 dist-test-slave-xd2q krb5kdc[9397](info): commencing operation
krb5kdc: starting...
W20260619 10:25:46.703495 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:46 dist-test-slave-xd2q krb5kdc[9397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:46.814081 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46327
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:46.891637 9410 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:46.891844 9410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:46.891871 9410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:46.891911 9410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:46.892957 9410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:46.893020 9410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:46.893064 9410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:46.893105 9410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:46.893127 9410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:46.894595 9410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:46.894850 9410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:46.895044 9410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:46.900564 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:46.900522 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:46.900532 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:46.900878 9410 server_base.cc:1034] running on GCE node
I20260619 10:25:46.901100 9410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:46.901396 9410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:46.902566 9410 hybrid_clock.cc:648] HybridClock initialized: now 1781864746902564 us; error 33 us; skew 500 ppm
Jun 19 10:25:46 dist-test-slave-xd2q krb5kdc[9397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:46.904783 9410 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:46.905181 9410 webserver.cc:460] Webserver started at http://127.19.65.190:35083/ using document root <none> and password file <none>
I20260619 10:25:46.905447 9410 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:46.905510 9410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:46.905653 9410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:46.906635 9410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "af5fd9ebe6f742279bbb82225a2b5eb7"
format_stamp: "Formatted at 2026-06-19 10:25:46 on dist-test-slave-xd2q"
server_key: "11aaebae1f608713857db479231f6788"
server_key_iv: "8fe99491b7ea84714b72c5b278e58a03"
server_key_version: "encryptionkey@0"
I20260619 10:25:46.906975 9410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "af5fd9ebe6f742279bbb82225a2b5eb7"
format_stamp: "Formatted at 2026-06-19 10:25:46 on dist-test-slave-xd2q"
server_key: "11aaebae1f608713857db479231f6788"
server_key_iv: "8fe99491b7ea84714b72c5b278e58a03"
server_key_version: "encryptionkey@0"
I20260619 10:25:46.908264 9410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:46.908989 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:46.909149 9410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:46.909258 9410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "af5fd9ebe6f742279bbb82225a2b5eb7"
format_stamp: "Formatted at 2026-06-19 10:25:46 on dist-test-slave-xd2q"
server_key: "11aaebae1f608713857db479231f6788"
server_key_iv: "8fe99491b7ea84714b72c5b278e58a03"
server_key_version: "encryptionkey@0"
I20260619 10:25:46.909323 9410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:46.935173 9410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:46.937558 9410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:46.937711 9410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:46.941545 9410 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:46327
I20260619 10:25:46.941574 9477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:46327 every 8 connection(s)
I20260619 10:25:46.941874 9410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:46.942765 9478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:46.944253 9478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:46.948313 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9410
I20260619 10:25:46.948427 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:46.948611 19718 external_mini_cluster.cc:1428] Setting key 3b80c184354aad39af579e5309354da2
Jun 19 10:25:46 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:46.953447 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:46.944788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46327 (local address 127.0.0.1:38386)
0619 10:25:46.944937 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:46.944972 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:46.945430 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:46.945677 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:46.945679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:46.945891 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:46.946043 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:46.946047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:46.946968 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:46.946968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:46.947724 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:46.947726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:46.948150 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:46.948150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:46.948198 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:46.948646 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:46.948654 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:46.950079 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:46.952053 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:46.952060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:46.952101 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:46.952293 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:46.952545 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:46.952546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:46.952546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:46.952621 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:46.953034 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:46.953038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:46.953095 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 10:25:46.953159 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:25:46.953268 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260619 10:25:46.953554 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:46.944841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.19.65.190:46327)
0619 10:25:46.944990 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:46.944993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:46.945038 (+ 45us) server_negotiation.cc:407] Connection header received
0619 10:25:46.945555 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:46.945557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:46.945572 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:46.945646 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:46.946133 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:46.946899 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:46.947790 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:46.948067 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:46.948180 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:46.950162 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:46.950168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:46.950169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:46.950174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:46.951963 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:46.952376 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:46.952379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:46.952379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:46.952478 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:46.952684 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:46.952685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:46.952686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:46.952966 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:46.953057 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:46.953218 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 10:25:46.953377 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260619 10:25:46.954347 9478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:46.954748 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:46.954639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58322)
0619 10:25:46.954685 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:46.954731 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:25:46.954865 9426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
I20260619 10:25:46.955230 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:46.955147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58324)
0619 10:25:46.955185 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:46.955207 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:25:46.955360 9478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:38993: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.65.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
I20260619 10:25:46.971925 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:38993
--webserver_interface=127.19.65.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:25:47.004554 9478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } attempt: 1
I20260619 10:25:47.004987 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.004872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58332)
0619 10:25:47.004924 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.004959 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:25:47.005332 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.005241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58336)
0619 10:25:47.005283 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.005309 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:25:47.005420 9478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:38993: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
W20260619 10:25:47.047492 9485 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:47.047664 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:47.047680 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:47.047732 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:47.048786 9485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:47.048825 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:47.048837 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:47.048849 9485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:47.048861 9485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:47.050271 9485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:38993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.65.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:47.050488 9485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:47.050705 9485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:47.055382 9493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:47.055392 9490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:47.055388 9491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:47.055687 9485 server_base.cc:1034] running on GCE node
I20260619 10:25:47.055935 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:47.056206 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:47.057363 9485 hybrid_clock.cc:648] HybridClock initialized: now 1781864747057357 us; error 32 us; skew 500 ppm
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:47.059880 9485 init.cc:377] Logged in from keytab as kudu/127.19.65.189@KRBTEST.COM (short username kudu)
I20260619 10:25:47.060323 9485 webserver.cc:460] Webserver started at http://127.19.65.189:44313/ using document root <none> and password file <none>
I20260619 10:25:47.060559 9485 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:47.060598 9485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:47.060739 9485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:47.061661 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data/instance:
uuid: "f9747f1d1ebb453a95933da7532e6cf5"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "a84cb20d03130a7e01938c7647a472d8"
server_key_iv: "958fa93f2d8639f3ee554fa75bf99d3b"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.061996 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal/instance:
uuid: "f9747f1d1ebb453a95933da7532e6cf5"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "a84cb20d03130a7e01938c7647a472d8"
server_key_iv: "958fa93f2d8639f3ee554fa75bf99d3b"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.063270 9485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:47.064083 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:47.064276 9485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:47.064373 9485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "f9747f1d1ebb453a95933da7532e6cf5"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "a84cb20d03130a7e01938c7647a472d8"
server_key_iv: "958fa93f2d8639f3ee554fa75bf99d3b"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.064456 9485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:47.073553 9478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } attempt: 2
I20260619 10:25:47.073961 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.073814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58352)
0619 10:25:47.073879 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.073927 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 10:25:47.074333 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.074239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58368)
0619 10:25:47.074285 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.074311 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 10:25:47.074437 9478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:38993: Network error: Client connection negotiation failed: client connection to 127.19.65.189:38993: connect: Connection refused (error 111)
I20260619 10:25:47.075161 9485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:47.076898 9485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:47.077030 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:47.080617 9485 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:38993
I20260619 10:25:47.080644 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.189:38993 every 8 connection(s)
I20260619 10:25:47.080950 9485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
I20260619 10:25:47.081856 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:47.083186 9553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:47.085878 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9485
I20260619 10:25:47.085975 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-1/wal/instance
I20260619 10:25:47.086149 19718 external_mini_cluster.cc:1428] Setting key 82669827293920542bb9a65c6d8e58f2
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:47.092038 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.083848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46327 (local address 127.0.0.1:38402)
0619 10:25:47.084025 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.084056 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.084466 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.084661 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.084663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.084876 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.085042 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.085045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.085648 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.085649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.086524 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.086527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.087025 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.087026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.087119 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.087591 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.087599 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.088935 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.090769 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.090775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.090776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.090969 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.091181 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.091181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.091181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.091277 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.091601 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.091606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.091727 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.091802 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.091906 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260619 10:25:47.092039 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.083850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.19.65.190:46327)
0619 10:25:47.083907 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.083909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.084131 (+ 222us) server_negotiation.cc:407] Connection header received
0619 10:25:47.084545 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.084546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.084557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.084611 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.085122 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.085572 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.086617 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.086938 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.087008 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.089023 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.089025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.089025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.089032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.090692 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.091042 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.091043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.091044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.091114 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.091383 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.091384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.091385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.091529 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.091598 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.091859 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.091941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:25:47.092635 9553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.65.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.188@KRBTEST.COM" created.
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:25:47.100768 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.092916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.19.65.189:38993)
0619 10:25:47.093058 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.093062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.093085 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:25:47.093115 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.093117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.093131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.093214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.093466 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.094590 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.095197 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.095587 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.095649 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.097620 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.097627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.097628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.097633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.099462 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.099907 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.099909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.099910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.099957 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.100188 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.100189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.100198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.100331 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.100387 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.100554 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.100680 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260619 10:25:47.100778 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.092853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58374)
0619 10:25:47.092896 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.092907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.092944 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.093223 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.093224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.093353 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.093394 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.093396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.094679 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.094680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.095130 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.095132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.095657 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.095658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.095725 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.096023 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.096026 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.097529 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.099542 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.099543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.099544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.099806 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.100032 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.100033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.100034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.100122 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.100383 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.100385 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.100438 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.100524 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.100635 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:25:47.101478 9553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:47.101821 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.101710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:33992)
0619 10:25:47.101755 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.101794 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:25:47.101994 9504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
I20260619 10:25:47.102535 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.102459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34008)
0619 10:25:47.102495 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.102518 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:25:47.102658 9553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:41695: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
I20260619 10:25:47.112056 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:41695
--webserver_interface=127.19.65.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:25:47.151876 9553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } attempt: 1
I20260619 10:25:47.152370 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.152206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34014)
0619 10:25:47.152307 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.152344 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 10:25:47.152683 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.152596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34028)
0619 10:25:47.152638 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.152661 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:25:47.152766 9553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:41695: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
I20260619 10:25:47.165563 9478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } attempt: 3
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:25:47.172643 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.165828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58388)
0619 10:25:47.165880 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.165891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.165937 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.166107 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.166108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.166272 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.166332 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.166333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.167099 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.167100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.167657 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.167660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.168218 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.168219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.168278 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.168714 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.168719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.170055 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.171741 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.171742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.171742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.171899 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.172102 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.172102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.172102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.172171 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.172426 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.172427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.172462 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.172512 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.172581 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:25:47.172730 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.165925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.19.65.189:38993)
0619 10:25:47.165987 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.165989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.165999 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:47.166029 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.166030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.166040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.166101 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.166413 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.167011 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.167752 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.168096 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.168208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.170165 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.170166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.170167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.170175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.171667 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.171967 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.171968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.171969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.172032 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.172231 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.172232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.172232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.172365 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.172431 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.172534 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.172622 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:25:47.173170 9478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:47.173540 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.173441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34036)
0619 10:25:47.173487 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.173512 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 10:25:47.173911 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.173851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34050)
0619 10:25:47.173876 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.173897 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 10:25:47.174011 9478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:41695: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
W20260619 10:25:47.191386 9560 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:47.191576 9560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:47.191592 9560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:47.191645 9560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:47.192771 9560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:47.192816 9560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:47.192830 9560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:47.192878 9560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:47.192898 9560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:47.194358 9560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:46327,127.19.65.189:38993,127.19.65.188:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.65.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:47.194594 9560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:47.194823 9560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:47.199414 9565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:47.199421 9566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:47.199445 9560 server_base.cc:1034] running on GCE node
W20260619 10:25:47.199429 9568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:47.199895 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:47.200204 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:47.201361 9560 hybrid_clock.cc:648] HybridClock initialized: now 1781864747201351 us; error 36 us; skew 500 ppm
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:47.203615 9560 init.cc:377] Logged in from keytab as kudu/127.19.65.188@KRBTEST.COM (short username kudu)
I20260619 10:25:47.204079 9560 webserver.cc:460] Webserver started at http://127.19.65.188:38533/ using document root <none> and password file <none>
I20260619 10:25:47.204263 9560 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:47.204303 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:47.204389 9560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:47.205134 9478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } attempt: 1
I20260619 10:25:47.205348 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data/instance:
uuid: "14162a35875c46b0852f64af9afec0cf"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "c68c0f1ea0a5ed401f632682229b8d87"
server_key_iv: "1020a1b8c277cfbebc286ee09388be8c"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.205473 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.205362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34052)
0619 10:25:47.205417 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.205450 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:25:47.205703 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal/instance:
uuid: "14162a35875c46b0852f64af9afec0cf"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "c68c0f1ea0a5ed401f632682229b8d87"
server_key_iv: "1020a1b8c277cfbebc286ee09388be8c"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.205788 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.205694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34060)
0619 10:25:47.205737 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.205761 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:25:47.205901 9478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:41695: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
I20260619 10:25:47.207022 9560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:25:47.207789 9575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:47.207971 9560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:47.208038 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "14162a35875c46b0852f64af9afec0cf"
format_stamp: "Formatted at 2026-06-19 10:25:47 on dist-test-slave-xd2q"
server_key: "c68c0f1ea0a5ed401f632682229b8d87"
server_key_iv: "1020a1b8c277cfbebc286ee09388be8c"
server_key_version: "encryptionkey@0"
I20260619 10:25:47.208117 9560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:47.216749 9560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:47.218343 9560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:47.218483 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:47.220902 9553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } attempt: 2
I20260619 10:25:47.221267 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.221151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34076)
0619 10:25:47.221207 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.221234 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 10:25:47.221621 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.221482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34078)
0619 10:25:47.221546 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.221585 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 10:25:47.221776 9553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:41695: Network error: Client connection negotiation failed: client connection to 127.19.65.188:41695: connect: Connection refused (error 111)
I20260619 10:25:47.222656 9560 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:41695
I20260619 10:25:47.222684 9627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.188:41695 every 8 connection(s)
I20260619 10:25:47.222980 9560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
I20260619 10:25:47.223884 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:47.225155 9628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:47.226138 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9560
I20260619 10:25:47.226213 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864489541676-19718-0/minicluster-data/master-2/wal/instance
I20260619 10:25:47.226375 19718 external_mini_cluster.cc:1428] Setting key eca625348a8fc76a35490ca808b1a7ad
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:47.234084 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.225765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.19.65.190:46327)
0619 10:25:47.225818 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.225820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.226008 (+ 188us) server_negotiation.cc:407] Connection header received
0619 10:25:47.226517 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.226518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.226530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.226613 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.227270 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.227719 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.228619 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.228798 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.228868 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.231182 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.231183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.231183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.231191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.232729 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.233175 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.233176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.233176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.233229 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.233511 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.233511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.233512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.233607 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.233655 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.233969 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.234028 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:25:47.234315 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.225721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:46327 (local address 127.0.0.1:38416)
0619 10:25:47.225876 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.225912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.226395 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.226678 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.226686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.226997 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.227150 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.227154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.227793 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.227794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.228519 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.228521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.228888 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.228889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.228997 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.229432 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.229439 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.231077 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.232799 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.232806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.232808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.233097 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.233289 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.233290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.233290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.233443 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.233671 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.233676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.233826 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.233951 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.234087 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":16,"thread_start_us":89,"threads_started":1}
I20260619 10:25:47.235172 9628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:47.235375 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.226778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.19.65.188:41695)
0619 10:25:47.226929 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.226933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.226958 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:25:47.227011 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.227013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.227035 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.227094 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.227757 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.228540 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.229221 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.229502 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.229574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.232160 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.232167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.232168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.232175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.233908 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.234292 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.234295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.234296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.234353 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.234747 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.234748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.234748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.234868 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.235021 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.235168 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.235240 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:47.242048 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.235500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.19.65.189:38993)
0619 10:25:47.235547 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.235549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.235683 (+ 134us) server_negotiation.cc:407] Connection header received
0619 10:25:47.235749 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.235750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.235778 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.235833 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.236157 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.236717 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.237285 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.237481 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.237559 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.239276 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.239277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.239277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.239284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.240826 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.241316 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.241317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.241317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.241370 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.241572 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.241573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.241574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.241693 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.241763 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.241901 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.241964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:25:47.242048 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.235450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:38993 (local address 127.0.0.1:58398)
0619 10:25:47.235633 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.235643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.235693 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.235846 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.235847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.235984 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.236057 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.236059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.236788 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.236789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.237209 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.237210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.237559 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.237560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.237631 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.238033 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.238037 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.239195 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.241086 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.241087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.241087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.241239 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.241446 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.241446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.241447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.241501 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.241757 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.241757 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.241797 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.241878 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.241961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 10:25:47.242746 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.236072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.19.65.190:46327)
0619 10:25:47.236110 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.236112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.236136 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:25:47.236155 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.236156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.236164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.236201 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.236744 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.237128 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.237713 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.237913 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.237982 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.240223 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.240225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.240225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.240232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.241565 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.241944 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.241946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.241946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.242005 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.242233 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.242234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.242235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.242371 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.242513 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.242553 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.242643 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:47.242835 9628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:25:47.249665 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.243117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34102)
0619 10:25:47.243148 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.243158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.243191 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.243457 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.243458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.243589 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.243611 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.243612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.244223 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.244223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.244635 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.244636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.245012 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.245013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.245060 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.245447 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.245454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.246791 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.248714 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.248715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.248716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.248856 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.249037 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.249038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.249038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.249080 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.249313 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.249313 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.249358 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.249422 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.249495 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 10:25:47.249675 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.243149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.19.65.188:41695)
0619 10:25:47.243364 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.243366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.243374 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:47.243392 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.243393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.243401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.243455 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.243683 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.244135 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.244713 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.244929 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.245012 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.246978 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.246979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.246979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.246985 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.248624 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.248921 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.248922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.248922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.248974 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.249156 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.249157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.249157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.249253 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.249297 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.249456 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.249527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 10:25:47.250370 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.243393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.19.65.189:38993)
0619 10:25:47.243422 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.243423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.243430 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:47.243467 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.243468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.243476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.243523 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.243959 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.244363 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.244973 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.245191 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.245257 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.247580 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.247582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.247582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.247589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.249061 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.249387 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.249387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.249387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.249448 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.249642 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.249643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.249643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.249887 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.250154 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.250203 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.250268 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:25:47.251364 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: Bootstrap starting.
I20260619 10:25:47.252508 9628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:47.252697 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:47.252852 9628 log.cc:824] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:47.253641 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: No bootstrap required, opened a new log
I20260619 10:25:47.253731 9452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38422
I20260619 10:25:47.255110 9628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.255230 9628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14162a35875c46b0852f64af9afec0cf, State: Initialized, Role: FOLLOWER
I20260619 10:25:47.255338 9628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.255657 9628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:47.255640 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.255743 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:25:47.257423 9645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:47.257476 9645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:47.257745 9560 master_runner.cc:186] Error getting master registration for 127.19.65.190:46327: OK, instance_id { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" instance_seqno: 1781864746940542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:25:47.259294 9560 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:25:47.281049 9478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } attempt: 2
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864746, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:47.288066 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.281577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.19.65.188:41695)
0619 10:25:47.281661 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.281663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.281672 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:47.281701 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.281702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.281709 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.281790 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.282078 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.282522 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.283213 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.283467 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.283550 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.285551 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.285552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.285553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.285560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.287057 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.287451 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.287452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.287452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.287495 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.287657 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.287657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.287657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.287772 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.287816 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.287953 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.288007 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 10:25:47.288121 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.281447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34118)
0619 10:25:47.281503 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.281514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.281554 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.281798 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.281800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.281957 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.281998 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.281999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.282610 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.282611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.283097 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.283099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.283549 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.283550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.283627 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.284073 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.284082 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.285452 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.287119 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.287120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.287120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.287365 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.287543 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.287544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.287544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.287603 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.287823 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.287824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.287870 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.287939 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.288025 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 10:25:47.289862 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: Bootstrap starting.
I20260619 10:25:47.290648 9478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:47.290938 9478 log.cc:824] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:47.291860 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: No bootstrap required, opened a new log
I20260619 10:25:47.293190 9478 raft_consensus.cc:348] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.293344 9478 raft_consensus.cc:729] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5fd9ebe6f742279bbb82225a2b5eb7, State: Initialized, Role: FOLLOWER
I20260619 10:25:47.293427 9478 consensus_queue.cc:260] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.293728 9478 sys_catalog.cc:564] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:47.293684 9648 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.293769 9452 raft_consensus.cc:482] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 10:25:47.293793 9648 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.293826 9452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:47.294495 9452 raft_consensus.cc:504] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.294724 9452 leader_election.cc:288] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [CANDIDATE]: Term 1 election: Requested vote from peers f9747f1d1ebb453a95933da7532e6cf5 (127.19.65.189:38993), 14162a35875c46b0852f64af9afec0cf (127.19.65.188:41695)
I20260619 10:25:47.295246 9602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14162a35875c46b0852f64af9afec0cf"
I20260619 10:25:47.295434 9602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [term 0 FOLLOWER]: Advancing to term 1
W20260619 10:25:47.296325 9659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:47.296370 9659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:47.296408 9602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af5fd9ebe6f742279bbb82225a2b5eb7 in term 1.
I20260619 10:25:47.296651 9429 leader_election.cc:302] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14162a35875c46b0852f64af9afec0cf, af5fd9ebe6f742279bbb82225a2b5eb7; no voters:
I20260619 10:25:47.296767 9648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:47.296907 9648 raft_consensus.cc:686] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [term 1 LEADER]: Becoming Leader. State: Replica: af5fd9ebe6f742279bbb82225a2b5eb7, State: Running, Role: LEADER
I20260619 10:25:47.296972 9648 consensus_queue.cc:237] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.297377 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af5fd9ebe6f742279bbb82225a2b5eb7. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.297490 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:47.298693 9667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:47.296716 9410 master_runner.cc:186] Error getting master registration for 127.19.65.189:38993: OK, instance_id { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" instance_seqno: 1781864747079676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:25:47.298866 9410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:25:47.308388 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.298159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.19.65.190:46327)
0619 10:25:47.298193 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.298194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.299082 (+ 888us) server_negotiation.cc:407] Connection header received
0619 10:25:47.299184 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.299185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.299194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.299220 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.299968 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.300595 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.302487 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.302786 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.302822 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.305153 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.305153 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.305154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.305159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.307012 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.307474 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.307476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.307476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.307537 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.307777 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.307779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.307779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.307917 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.308005 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.308216 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.308284 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:47.298868 9667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:47.309948 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.297904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.19.65.189:38993)
0619 10:25:47.297941 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.297943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.298473 (+ 530us) server_negotiation.cc:407] Connection header received
0619 10:25:47.298606 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.298607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.298654 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.298685 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.299863 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.300571 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.302463 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.302766 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.302822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.305132 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.305133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.305134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.305150 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.308461 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.309025 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.309026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.309026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.309088 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.309365 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.309365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.309366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.309497 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.309589 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.309772 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.309845 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:47.311502 9602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [term 1 FOLLOWER]: Refusing update from remote peer af5fd9ebe6f742279bbb82225a2b5eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:47.311856 9661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:47.313271 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader af5fd9ebe6f742279bbb82225a2b5eb7. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.313375 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.313431 9553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } attempt: 3
I20260619 10:25:47.316694 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.298566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.19.65.188:41695)
0619 10:25:47.298598 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.298599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.303403 (+ 4804us) server_negotiation.cc:407] Connection header received
0619 10:25:47.305675 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.305676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.305686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.305719 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.306354 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.307089 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.310952 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.311239 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.311296 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.313709 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.313711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.313711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.313718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.315476 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.315858 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.315859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.315859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.315908 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.316094 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.316095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.316095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.316231 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.316327 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.316524 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.316594 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:25:47.318234 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.318326 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:47.318960 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.319042 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.319970 9672 mvcc.cc:205] Tried to move back new op lower bound from 7298518004985143296 to 7298518004928991232. Current Snapshot: MvccSnapshot[applied={T|T < 7298518004985143296}]
Jun 19 10:25:47 dist-test-slave-xd2q krb5kdc[9397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864747, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:47.320755 9667 catalog_manager.cc:1347] Generated new cluster ID: 8177c73d91f54bf98cc11174623d6de9
I20260619 10:25:47.320803 9667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:47.329542 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.313655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:41695 (local address 127.0.0.1:34138)
0619 10:25:47.313730 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:47.313739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:47.313773 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:47.314277 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:47.314278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:47.314447 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:47.314489 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.314491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.316209 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.316224 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.317008 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:47.317010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.318458 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.318459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:47.318521 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.318917 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:47.318921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:47.320391 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:47.328563 (+ 8172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.328564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.328564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.328744 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.328910 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:47.328911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:47.328911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:47.328982 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:47.329233 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:47.329233 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:47.329275 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:25:47.329347 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:47.329436 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 10:25:47.329550 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:47.313834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.19.65.188:41695)
0619 10:25:47.314028 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.314031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.314038 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:47.314096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.314097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.314116 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.314579 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.314646 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.316061 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.317713 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:47.318322 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:47.320126 (+ 1804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:47.320488 (+ 362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:47.320489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:47.320490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:47.320496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:47.328479 (+ 7983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.328806 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.328807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.328807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.328852 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:47.329046 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:47.329046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:47.329047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:47.329160 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:47.329225 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:47.329376 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 10:25:47.329446 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260619 10:25:47.330915 9553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: Bootstrap starting.
I20260619 10:25:47.331586 9553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:47.331875 9553 log.cc:824] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:47.334800 9667 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:47.336205 9667 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:47.339905 9553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: No bootstrap required, opened a new log
I20260619 10:25:47.341852 9553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.341984 9553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9747f1d1ebb453a95933da7532e6cf5, State: Initialized, Role: FOLLOWER
I20260619 10:25:47.342051 9553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } }
I20260619 10:25:47.342329 9553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:47.342401 9526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:47.343217 9526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [term 1 FOLLOWER]: Refusing update from remote peer af5fd9ebe6f742279bbb82225a2b5eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:47.343412 9527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9747f1d1ebb453a95933da7532e6cf5"
I20260619 10:25:47.344185 9527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af5fd9ebe6f742279bbb82225a2b5eb7 in term 1.
I20260619 10:25:47.344636 9661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:47.348357 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.348445 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:47.348834 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.348908 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.348950 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af5fd9ebe6f742279bbb82225a2b5eb7. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.348986 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.349012 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5fd9ebe6f742279bbb82225a2b5eb7" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 46327 } } peers { permanent_uuid: "f9747f1d1ebb453a95933da7532e6cf5" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 38993 } } peers { permanent_uuid: "14162a35875c46b0852f64af9afec0cf" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 41695 } } }
I20260619 10:25:47.349046 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:47.349471 9691 mvcc.cc:205] Tried to move back new op lower bound from 7298518005079601152 to 7298518004928991232. Current Snapshot: MvccSnapshot[applied={T|T < 7298518004985143296}]
I20260619 10:25:47.349478 9667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af5fd9ebe6f742279bbb82225a2b5eb7: Generated new TSK 0
I20260619 10:25:47.349713 9667 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:25:47.350121 9689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:47.350507 9689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 10:25:47.367197 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.365971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.19.65.188:41695)
0619 10:25:47.366009 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.366011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.366556 (+ 545us) server_negotiation.cc:407] Connection header received
0619 10:25:47.366693 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.366694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.366707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.366736 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.367120 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34142: BlockingRecv error: recv got EOF from 127.0.0.1:34142 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 10:25:47.367920 9481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.366290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.19.65.190:46327)
0619 10:25:47.366325 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.366326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.367427 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 10:25:47.367513 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.367514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.367524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.367551 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.367871 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38446: BlockingRecv error: recv got EOF from 127.0.0.1:38446 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:47.368835 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.368205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.19.65.189:38993)
0619 10:25:47.368239 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.368241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.368352 (+ 111us) server_negotiation.cc:407] Connection header received
0619 10:25:47.368433 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.368434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.368443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.368470 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.368785 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58436: BlockingRecv error: recv got EOF from 127.0.0.1:58436 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 10:25:47.370850 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.370172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.19.65.189:38993)
0619 10:25:47.370204 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.370205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.370395 (+ 190us) server_negotiation.cc:407] Connection header received
0619 10:25:47.370486 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.370487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.370501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.370528 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.370812 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58444: BlockingRecv error: recv got EOF from 127.0.0.1:58444 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:47.373574 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.372900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.19.65.189:38993)
0619 10:25:47.372928 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.372929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.373107 (+ 178us) server_negotiation.cc:407] Connection header received
0619 10:25:47.373190 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.373190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.373199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.373226 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.373525 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58452: BlockingRecv error: recv got EOF from 127.0.0.1:58452 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 10:25:47.377193 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.376558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.19.65.189:38993)
0619 10:25:47.376589 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.376602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.376777 (+ 175us) server_negotiation.cc:407] Connection header received
0619 10:25:47.376860 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.376861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.376869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.376895 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.377153 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58466: BlockingRecv error: recv got EOF from 127.0.0.1:58466 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 10:25:47.382799 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.382180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.19.65.189:38993)
0619 10:25:47.382207 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.382208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.382386 (+ 178us) server_negotiation.cc:407] Connection header received
0619 10:25:47.382465 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.382466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.382473 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.382498 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.382763 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58470: BlockingRecv error: recv got EOF from 127.0.0.1:58470 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 10:25:47.392614 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.391941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.19.65.189:38993)
0619 10:25:47.391972 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.391973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.392163 (+ 190us) server_negotiation.cc:407] Connection header received
0619 10:25:47.392250 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.392251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.392260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.392286 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.392560 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58474: BlockingRecv error: recv got EOF from 127.0.0.1:58474 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 10:25:47.410606 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.409777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.19.65.189:38993)
0619 10:25:47.409810 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.409812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.410015 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:25:47.410143 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.410144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.410155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.410182 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.410551 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58484: BlockingRecv error: recv got EOF from 127.0.0.1:58484 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:47.444933 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.444003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.19.65.189:38993)
0619 10:25:47.444062 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.444064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.444318 (+ 254us) server_negotiation.cc:407] Connection header received
0619 10:25:47.444420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.444421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.444434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.444462 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.444871 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58492: BlockingRecv error: recv got EOF from 127.0.0.1:58492 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 10:25:47.511076 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.510273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.19.65.189:38993)
0619 10:25:47.510348 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.510350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.510487 (+ 137us) server_negotiation.cc:407] Connection header received
0619 10:25:47.510555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.510556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.510581 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.510631 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.511012 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58494: BlockingRecv error: recv got EOF from 127.0.0.1:58494 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 10:25:47.640847 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.640127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.19.65.189:38993)
0619 10:25:47.640209 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.640211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.640298 (+ 87us) server_negotiation.cc:407] Connection header received
0619 10:25:47.640394 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.640395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.640407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.640465 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.640783 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58510: BlockingRecv error: recv got EOF from 127.0.0.1:58510 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 10:25:47.898562 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:47.897917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.19.65.189:38993)
0619 10:25:47.897967 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:47.897968 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:47.898039 (+ 71us) server_negotiation.cc:407] Connection header received
0619 10:25:47.898083 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:47.898084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:47.898094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:47.898142 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:47.898477 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58514: BlockingRecv error: recv got EOF from 127.0.0.1:58514 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:48.257872 9645 catalog_manager.cc:1259] Loaded cluster ID: 8177c73d91f54bf98cc11174623d6de9
I20260619 10:25:48.257930 9645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: loading cluster ID for follower catalog manager: success
I20260619 10:25:48.258658 9645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: acquiring CA information for follower catalog manager: success
I20260619 10:25:48.258893 9645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14162a35875c46b0852f64af9afec0cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:25:48.351030 9689 catalog_manager.cc:1259] Loaded cluster ID: 8177c73d91f54bf98cc11174623d6de9
I20260619 10:25:48.351121 9689 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: loading cluster ID for follower catalog manager: success
I20260619 10:25:48.351884 9689 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: acquiring CA information for follower catalog manager: success
I20260619 10:25:48.352110 9689 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9747f1d1ebb453a95933da7532e6cf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:25:48.415107 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:48.411762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.19.65.189:38993)
0619 10:25:48.411894 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:48.411896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:48.411989 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:25:48.412082 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:48.412083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:48.412158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:48.412261 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:48.412438 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.412925 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.414226 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.414540 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.414591 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:48.414808 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:48.414909 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:48.414961 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:48.415031 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:48.415047 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260619 10:25:48.419632 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:48.416846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.19.65.188:41695)
0619 10:25:48.416956 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:48.416958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:48.417022 (+ 64us) server_negotiation.cc:407] Connection header received
0619 10:25:48.417106 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:48.417106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:48.417148 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:48.417224 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:48.417345 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.417806 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.418876 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.419072 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.419120 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:48.419322 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:48.419409 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:48.419460 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:48.419531 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 10:25:48.419547 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260619 10:25:48.424461 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:48.421497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.19.65.189:38993)
0619 10:25:48.421529 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:48.421531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:48.421613 (+ 82us) server_negotiation.cc:407] Connection header received
0619 10:25:48.421718 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:48.421719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:48.421729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:48.421755 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:48.421919 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.422563 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.423663 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.423951 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.423987 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:48.424182 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:48.424215 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:48.424244 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:48.424359 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 10:25:48.424395 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:48.425040 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:48.421394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.19.65.190:46327)
0619 10:25:48.421494 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:48.421496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:48.421644 (+ 148us) server_negotiation.cc:407] Connection header received
0619 10:25:48.421742 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:48.421743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:48.421786 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:48.421875 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:48.421994 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.422692 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.423719 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.424036 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.424268 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:48.424337 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:48.424443 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:48.424913 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:48.424931 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 10:25:48.424956 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
I20260619 10:25:48.425280 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:48.421736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.19.65.188:41695)
0619 10:25:48.421765 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:48.421766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:48.422074 (+ 308us) server_negotiation.cc:407] Connection header received
0619 10:25:48.422160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:48.422161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:48.422171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:48.422197 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:48.422374 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.423140 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.424370 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:48.424763 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:48.424808 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:48.425031 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:48.425063 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:48.425117 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:48.425187 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:48.425209 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:25:48.426643 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9410
I20260619 10:25:48.431645 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9485
I20260619 10:25:48.436507 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9560
2026-06-19T10:25:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2748 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:48 dist-test-slave-xd2q krb5kdc[9779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:48 dist-test-slave-xd2q krb5kdc[9779](info): set up 2 sockets
Jun 19 10:25:48 dist-test-slave-xd2q krb5kdc[9779](info): commencing operation
krb5kdc: starting...
W20260619 10:25:49.451501 19718 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/test-user.keytab.
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.65.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
Entry for principal kudu/127.19.65.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab.
I20260619 10:25:49.559962 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33701
--webserver_interface=127.19.65.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:49.636412 9792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:49.636610 9792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:49.636627 9792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:49.636647 9792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:49.637681 9792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:49.637712 9792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:49.637727 9792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:49.637738 9792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:49.637749 9792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:49.639070 9792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.190.keytab
--principal=kudu/127.19.65.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.190:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.65.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:49.639238 9792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:49.639470 9792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:49.643977 9798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:49.643962 9797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:49.644153 9792 server_base.cc:1034] running on GCE node
W20260619 10:25:49.643961 9800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:49.644407 9792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:49.644678 9792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:49.645852 9792 hybrid_clock.cc:648] HybridClock initialized: now 1781864749645844 us; error 29 us; skew 500 ppm
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:49.648234 9792 init.cc:377] Logged in from keytab as kudu/127.19.65.190@KRBTEST.COM (short username kudu)
I20260619 10:25:49.648622 9792 webserver.cc:460] Webserver started at http://127.19.65.190:41603/ using document root <none> and password file <none>
I20260619 10:25:49.648835 9792 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:49.648901 9792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:49.649040 9792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:49.650023 9792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data/instance:
uuid: "52543dc79fc742d49708eb3ba81a08d3"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "f08ec569513dfd66cb4b2cc7bbd50bac"
server_key_iv: "f4c791145cd067b1ebff47b9bf67efbd"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.650377 9792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance:
uuid: "52543dc79fc742d49708eb3ba81a08d3"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "f08ec569513dfd66cb4b2cc7bbd50bac"
server_key_iv: "f4c791145cd067b1ebff47b9bf67efbd"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.651691 9792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:49.652390 9807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:49.652555 9792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:49.652657 9792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal
uuid: "52543dc79fc742d49708eb3ba81a08d3"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "f08ec569513dfd66cb4b2cc7bbd50bac"
server_key_iv: "f4c791145cd067b1ebff47b9bf67efbd"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.652734 9792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:49.687194 9792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:49.688957 9792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:49.689121 9792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:49.692896 9792 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.190:33701
I20260619 10:25:49.692929 9859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.190:33701 every 8 connection(s)
I20260619 10:25:49.693256 9792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:49.694031 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9792
I20260619 10:25:49.694134 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-0/wal/instance
I20260619 10:25:49.694329 19718 external_mini_cluster.cc:1428] Setting key daa4ef437b17d74ce16106ed91ff2186
I20260619 10:25:49.694428 9860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:49.696486 9860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.65.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.189@KRBTEST.COM" created.
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:49.707700 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.697209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.19.65.190:33701)
0619 10:25:49.697359 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.697368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.697478 (+ 110us) server_negotiation.cc:407] Connection header received
0619 10:25:49.698000 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.698006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.698022 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.698127 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.698638 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.699504 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.700362 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.700653 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.700778 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.704622 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.704630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.704630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.704638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.706161 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.706534 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.706536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.706537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.706618 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.706848 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.706849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.706849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.707043 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.707136 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.707332 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.707480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260619 10:25:49.707701 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.697168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33701 (local address 127.0.0.1:46344)
0619 10:25:49.697341 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.697379 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.697850 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.698145 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.698148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.698402 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.698538 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.698543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.699607 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.699608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.700267 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.700269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.700767 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.700768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.700830 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.701348 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.701358 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.704529 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.706235 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.706241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.706242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.706422 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.706701 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.706701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.706702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.706781 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.707116 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.707121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.707182 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.707283 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.707542 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":102,"thread_start_us":86,"threads_started":1}
I20260619 10:25:49.708340 9860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:49.708673 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.708563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54528)
0619 10:25:49.708609 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.708651 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:25:49.708813 9808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
I20260619 10:25:49.709218 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.709136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54536)
0619 10:25:49.709174 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.709196 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 10:25:49.709319 9860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:42889: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
Entry for principal kudu/127.19.65.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab.
I20260619 10:25:49.718118 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:42889
--webserver_interface=127.19.65.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:25:49.758539 9860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } attempt: 1
I20260619 10:25:49.759034 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.758869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54546)
0619 10:25:49.758953 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.758998 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 10:25:49.759457 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.759351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54562)
0619 10:25:49.759406 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.759435 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 10:25:49.759585 9860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:42889: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
W20260619 10:25:49.795790 9867 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:49.796005 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:49.796033 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:49.796087 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:49.797240 9867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:49.797302 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:49.797326 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:49.797364 9867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:49.797386 9867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:49.798892 9867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.189.keytab
--principal=kudu/127.19.65.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.189:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.65.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:49.799189 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:49.799427 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:49.803948 9867 server_base.cc:1034] running on GCE node
W20260619 10:25:49.803957 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:49.803963 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:49.804083 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:49.804422 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:49.804713 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:49.805876 9867 hybrid_clock.cc:648] HybridClock initialized: now 1781864749805862 us; error 25 us; skew 500 ppm
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:49.808248 9867 init.cc:377] Logged in from keytab as kudu/127.19.65.189@KRBTEST.COM (short username kudu)
I20260619 10:25:49.808590 9867 webserver.cc:460] Webserver started at http://127.19.65.189:41171/ using document root <none> and password file <none>
I20260619 10:25:49.808753 9867 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:49.808787 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:49.808864 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:49.809747 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data/instance:
uuid: "10a0b32065d04dc3a39787b5564acc4c"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "d97f0c55c70f80d253629359ae81ed18"
server_key_iv: "80c01e7f17a5d8ed72465747485f6ea6"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.809999 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal/instance:
uuid: "10a0b32065d04dc3a39787b5564acc4c"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "d97f0c55c70f80d253629359ae81ed18"
server_key_iv: "80c01e7f17a5d8ed72465747485f6ea6"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.811126 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:49.811904 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:49.812039 9867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:49.812111 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal
uuid: "10a0b32065d04dc3a39787b5564acc4c"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "d97f0c55c70f80d253629359ae81ed18"
server_key_iv: "80c01e7f17a5d8ed72465747485f6ea6"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.812160 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:49.826584 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:49.827729 9860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } attempt: 2
I20260619 10:25:49.828179 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.828027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54576)
0619 10:25:49.828088 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.828129 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 10:25:49.828487 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.828416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54590)
0619 10:25:49.828446 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.828468 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 10:25:49.828586 9860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.189:42889: Network error: Client connection negotiation failed: client connection to 127.19.65.189:42889: connect: Connection refused (error 111)
I20260619 10:25:49.828940 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:49.829082 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:49.832682 9867 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.189:42889
I20260619 10:25:49.832746 9934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.189:42889 every 8 connection(s)
I20260619 10:25:49.833040 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/data/info.pb
I20260619 10:25:49.833981 9935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:49.835392 9935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:49.842213 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9867
I20260619 10:25:49.842303 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-1/wal/instance
I20260619 10:25:49.842525 19718 external_mini_cluster.cc:1428] Setting key f355267fed25aaf87948b97384abc732
I20260619 10:25:49.843744 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.836010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.19.65.190:33701)
0619 10:25:49.836069 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.836070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.836312 (+ 242us) server_negotiation.cc:407] Connection header received
0619 10:25:49.836706 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.836707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.836730 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.836780 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.837305 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.837754 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.838682 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.838866 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.838917 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.840875 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.840876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.840876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.840883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.842282 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.842775 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.842776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.842776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.842841 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.843113 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.843114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.843114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.843221 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.843291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.843573 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.843652 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:25:49.843909 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.836014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33701 (local address 127.0.0.1:46358)
0619 10:25:49.836206 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.836236 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.836631 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.836848 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.836851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.837055 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.837238 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.837242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.837825 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.837826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.838605 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.838607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.838948 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.838949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.839050 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.839521 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.839529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.840781 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.842401 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.842407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.842409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.842698 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.842927 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.842928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.842928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.843040 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.843317 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.843324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.843447 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.843553 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.843696 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260619 10:25:49.844393 9935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.65.188@KRBTEST.COM; defaulting to no policy
I20260619 10:25:49.851667 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.844628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54606)
0619 10:25:49.844691 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.844700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.844742 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.845019 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.845019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.845171 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.845220 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.845222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.846009 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.846010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.846874 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.846876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.847339 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.847340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.847425 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.847746 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.847752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.848937 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.850614 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.850614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.850615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.850775 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.850988 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.850989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.850989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.851040 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.851362 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.851362 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.851398 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.851467 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.851554 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 10:25:49.851751 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.844725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.19.65.189:42889)
0619 10:25:49.844897 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.844900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.844915 (+ 15us) server_negotiation.cc:407] Connection header received
0619 10:25:49.844930 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.844931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.844941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.845019 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.845294 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.845920 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.846954 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.847237 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.847333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.849017 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.849024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.849025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.849031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.850523 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.850870 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.850872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.850873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.850924 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.851108 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.851109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.851110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.851264 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.851383 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.851496 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.851617 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260619 10:25:49.852298 9935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:49.852583 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.852475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58718)
0619 10:25:49.852518 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.852560 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.65.188@KRBTEST.COM" created.
W20260619 10:25:49.852681 9886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
I20260619 10:25:49.853036 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.852938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58724)
0619 10:25:49.852992 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.853017 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 10:25:49.853142 9935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:38625: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
Entry for principal kudu/127.19.65.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab.
I20260619 10:25:49.866393 19718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskadFLwT/build/release/bin/kudu
/tmp/dist-test-taskadFLwT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:38625
--webserver_interface=127.19.65.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.65.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:25:49.902377 9935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } attempt: 1
I20260619 10:25:49.902872 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.902743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58728)
0619 10:25:49.902799 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.902841 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:25:49.903239 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.903136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58732)
0619 10:25:49.903195 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.903221 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 10:25:49.903375 9935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:38625: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
I20260619 10:25:49.919713 9860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } attempt: 3
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:25:49.926213 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.920048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.19.65.189:42889)
0619 10:25:49.920106 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.920108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.920117 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:49.920163 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.920164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.920175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.920245 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.920543 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.921029 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.921729 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.921939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.922008 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.923745 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.923746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.923747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.923753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.925098 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.925431 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.925432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.925432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.925509 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.925711 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.925712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.925713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.925825 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.925900 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.926049 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.926123 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:25:49.926226 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.919973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54610)
0619 10:25:49.920026 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.920038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.920103 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.920251 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.920252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.920401 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.920465 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.920467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.921118 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.921119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.921648 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.921649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.922017 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.922018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.922091 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.922513 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.922517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.923652 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.925198 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.925199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.925199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.925361 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.925578 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.925579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.925580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.925642 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.925891 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.925892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.925936 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.926025 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.926123 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:25:49.926564 9860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:49.926896 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.926805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58736)
0619 10:25:49.926849 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.926874 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:25:49.927249 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.927193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58740)
0619 10:25:49.927218 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.927237 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 10:25:49.927353 9860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:38625: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
W20260619 10:25:49.944399 9942 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:25:49.944576 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:49.944592 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:49.944612 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:49.945634 9942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:49.945675 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:49.945689 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:49.945701 9942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:49.945713 9942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:49.947070 9942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.65.148:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.65.190:33701,127.19.65.189:42889,127.19.65.188:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/krb5kdc/kudu_127.19.65.188.keytab
--principal=kudu/127.19.65.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.65.188:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.65.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:49.947245 9942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:49.947450 9942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:49.951853 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:49.951860 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:49.951860 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:49.952354 9942 server_base.cc:1034] running on GCE node
I20260619 10:25:49.952569 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:49.952790 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:49.953922 9942 hybrid_clock.cc:648] HybridClock initialized: now 1781864749953898 us; error 37 us; skew 500 ppm
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:49.956424 9942 init.cc:377] Logged in from keytab as kudu/127.19.65.188@KRBTEST.COM (short username kudu)
I20260619 10:25:49.956892 9942 webserver.cc:460] Webserver started at http://127.19.65.188:35297/ using document root <none> and password file <none>
I20260619 10:25:49.957109 9942 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:49.957150 9942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:49.957284 9942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:49.958225 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data/instance:
uuid: "61b21736c6b141fc910071b21c42a7eb"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "e7bef577773513cae821efe667afbf40"
server_key_iv: "2af5391af4e4fce36a786d64f64044cb"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.958457 9860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } attempt: 1
I20260619 10:25:49.958567 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal/instance:
uuid: "61b21736c6b141fc910071b21c42a7eb"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "e7bef577773513cae821efe667afbf40"
server_key_iv: "2af5391af4e4fce36a786d64f64044cb"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.958753 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.958661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58752)
0619 10:25:49.958702 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.958733 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 10:25:49.959013 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.958937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58766)
0619 10:25:49.958973 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.958999 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 10:25:49.959090 9860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:38625: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
I20260619 10:25:49.960024 9942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:49.960805 9957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:49.960974 9942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:49.961058 9942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data,/tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal
uuid: "61b21736c6b141fc910071b21c42a7eb"
format_stamp: "Formatted at 2026-06-19 10:25:49 on dist-test-slave-xd2q"
server_key: "e7bef577773513cae821efe667afbf40"
server_key_iv: "2af5391af4e4fce36a786d64f64044cb"
server_key_version: "encryptionkey@0"
I20260619 10:25:49.961123 9942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:49.971535 9935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } attempt: 2
I20260619 10:25:49.971598 9942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:49.971845 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.971740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58780)
0619 10:25:49.971792 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.971819 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:25:49.972203 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.972087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58788)
0619 10:25:49.972147 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.972172 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 10:25:49.972298 9935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.65.188:38625: Network error: Client connection negotiation failed: client connection to 127.19.65.188:38625: connect: Connection refused (error 111)
I20260619 10:25:49.973531 9942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:49.973680 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:49.977337 9942 rpc_server.cc:307] RPC server started. Bound to: 127.19.65.188:38625
I20260619 10:25:49.977397 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.65.188:38625 every 8 connection(s)
I20260619 10:25:49.977715 9942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/data/info.pb
I20260619 10:25:49.978610 10010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:49.980180 19718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskadFLwT/build/release/bin/kudu as pid 9942
I20260619 10:25:49.980062 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:49.980293 19718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskadFLwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864489541676-19718-0/minicluster-data/master-2/wal/instance
I20260619 10:25:49.980476 19718 external_mini_cluster.cc:1428] Setting key cd94df5d5d1f39e0c20bc5cc4d85956a
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:49.989743 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.980658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.19.65.190:33701)
0619 10:25:49.980731 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.980733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.980967 (+ 234us) server_negotiation.cc:407] Connection header received
0619 10:25:49.981449 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.981450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.981460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.981514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.982227 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.982974 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.984135 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.984355 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.984420 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.986547 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.986548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.986548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.986556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.988247 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.988672 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.988673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.988673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.988744 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.989056 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.989056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.989057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.989195 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.989270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.989573 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.989654 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:25:49.989940 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.980952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.19.65.188:38625)
0619 10:25:49.981077 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.981081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.981112 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:25:49.981374 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.981383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.981402 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.981509 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.982235 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.983362 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.984300 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.984652 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.984735 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.987070 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.987077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.987077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.987084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.988759 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.989167 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.989170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.989170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.989222 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.989412 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.989413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.989413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.989552 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.989702 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.989780 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.989853 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260619 10:25:49.989948 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.980662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.190:33701 (local address 127.0.0.1:46372)
0619 10:25:49.980830 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.980876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.981360 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.981542 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.981544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.981866 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.982081 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.982087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.983051 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.983051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.984054 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.984057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.984449 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.984450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.984556 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.984979 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.984986 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.986457 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.988327 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.988332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.988334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.988598 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.988844 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.988844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.988845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.988988 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.989285 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.989292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.989442 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.989526 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.989678 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260619 10:25:49.990492 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.190@KRBTEST.COM
I20260619 10:25:49.997332 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.990716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.189:42889 (local address 127.0.0.1:54612)
0619 10:25:49.990815 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.990826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.990887 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.991026 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.991027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.991129 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.991152 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.991153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.991917 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.991918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.992444 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.992446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.992906 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.992907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.992958 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.993365 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:49.993369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:49.994651 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:49.996487 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.996488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.996488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.996639 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.996850 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:49.996851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:49.996851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:49.996897 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:49.997138 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:49.997138 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:49.997167 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 10:25:49.997224 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:25:49.997282 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 10:25:49.997422 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.990778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.19.65.189:42889)
0619 10:25:49.990818 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.990820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.990863 (+ 43us) server_negotiation.cc:407] Connection header received
0619 10:25:49.990963 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.990964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.990974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.991025 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.991239 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.991849 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.992517 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.992837 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.992890 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.994731 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.994732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.994733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.994739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.996402 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.996715 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.996716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.996716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.996772 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.996952 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.996953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.996953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.997078 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.997146 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.997251 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.997317 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:49.997893 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:25:49.998250 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.990831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.19.65.190:33701)
0619 10:25:49.990868 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.990870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.990877 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:49.990895 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.990896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.990905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.990957 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.991559 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.992055 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.992771 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.992986 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.993054 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:49.995648 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:49.995649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:49.995650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:49.995656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:49.997090 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.997457 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.997458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.997458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.997508 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:49.997702 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:49.997703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:49.997703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:49.997803 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:49.997960 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:49.997996 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 10:25:49.998062 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.188@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
Jun 19 10:25:49 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.65.189@KRBTEST.COM
I20260619 10:25:50.004961 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.998121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58796)
0619 10:25:49.998173 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:49.998181 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:49.998213 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:49.998446 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:49.998447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:49.998579 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:49.998599 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.998600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.999280 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.999281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:49.999786 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:49.999788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.000160 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.000161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:50.000206 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.000640 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:50.000646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:50.001985 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:50.003855 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.003856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.003856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.004009 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.004263 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.004263 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.004264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.004331 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.004625 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:50.004626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:50.004689 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 10:25:50.004768 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 10:25:50.004881 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 10:25:50.005074 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.998188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.19.65.188:38625)
0619 10:25:49.998225 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.998227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.998234 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:49.998253 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.998254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.998262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.998311 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.998683 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:49.999203 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:49.999877 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.000081 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.000161 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.002067 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.002069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.002069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.002077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.003776 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.004098 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.004099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.004099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.004161 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.004400 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.004401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.004401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.004555 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.004631 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.004813 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.004944 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":21}
I20260619 10:25:50.006510 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: Bootstrap starting.
I20260619 10:25:50.006758 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:49.998916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.19.65.189:42889)
0619 10:25:49.998954 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:49.998956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:49.998975 (+ 19us) server_negotiation.cc:407] Connection header received
0619 10:25:49.998992 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:49.998992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:49.999000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:49.999043 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:49.999574 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.000005 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.000857 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.001098 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.001164 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.003846 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.003847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.003847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.003853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.005568 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.005898 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.005899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.005900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.005952 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.006173 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.006174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.006174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.006287 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.006459 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.006503 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.006577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:50.007571 10010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:50.007899 10010 log.cc:824] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:50.008205 19718 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:25:50.008664 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: No bootstrap required, opened a new log
I20260619 10:25:50.009212 9834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "52543dc79fc742d49708eb3ba81a08d3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46378
I20260619 10:25:50.010718 10010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.010867 10010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b21736c6b141fc910071b21c42a7eb, State: Initialized, Role: FOLLOWER
I20260619 10:25:50.010948 10010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.011260 10010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:50.011231 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.011375 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:25:50.013105 10027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:50.013147 10027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:50.013507 9942 master_runner.cc:186] Error getting master registration for 127.19.65.190:33701: OK, instance_id { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" instance_seqno: 1781864749691781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:25:50.016547 9942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:25:50.034260 9860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } attempt: 2
Jun 19 10:25:50 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.190@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:50.043123 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.034741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58808)
0619 10:25:50.034812 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:50.034828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:50.034900 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:50.035105 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:50.035106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:50.035335 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:50.035389 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:50.035391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.036344 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.036345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:50.037180 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:50.037182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.037650 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.037651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:50.037719 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.038251 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:50.038257 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:50.039899 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:50.041892 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.041893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.041894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.042115 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.042336 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.042337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.042337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.042406 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.042783 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:50.042784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:50.042836 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 10:25:50.042908 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:50.043013 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 10:25:50.043454 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.034810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.19.65.188:38625)
0619 10:25:50.034861 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.034863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.034882 (+ 19us) server_negotiation.cc:407] Connection header received
0619 10:25:50.034971 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.034972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.035000 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.035099 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.035473 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.036248 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.037273 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.037571 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.037657 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.040011 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.040013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.040013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.040021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.041800 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.042198 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.042199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.042199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.042264 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.042473 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.042474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.042474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.042680 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.042784 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.043260 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.043363 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:25:50.045042 9860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: Bootstrap starting.
I20260619 10:25:50.045797 9860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:50.046093 9860 log.cc:824] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:50.047003 9860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: No bootstrap required, opened a new log
I20260619 10:25:50.048381 9860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.048527 9860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52543dc79fc742d49708eb3ba81a08d3, State: Initialized, Role: FOLLOWER
I20260619 10:25:50.048614 9860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.048949 9860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:50.048900 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.048976 9834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 10:25:50.049024 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.049225 9834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:50.050102 9834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
W20260619 10:25:50.050905 10041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:50.051010 10041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:50.051035 9984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52543dc79fc742d49708eb3ba81a08d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61b21736c6b141fc910071b21c42a7eb"
I20260619 10:25:50.051196 9984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:50.052274 9984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52543dc79fc742d49708eb3ba81a08d3 in term 1.
I20260619 10:25:50.052522 9834 leader_election.cc:288] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [CANDIDATE]: Term 1 election: Requested vote from peers 10a0b32065d04dc3a39787b5564acc4c (127.19.65.189:42889), 61b21736c6b141fc910071b21c42a7eb (127.19.65.188:38625)
I20260619 10:25:50.051903 9792 master_runner.cc:186] Error getting master registration for 127.19.65.189:42889: OK, instance_id { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" instance_seqno: 1781864749831724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:25:50.054060 9792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:25:50.054158 9811 leader_election.cc:302] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52543dc79fc742d49708eb3ba81a08d3, 61b21736c6b141fc910071b21c42a7eb; no voters:
I20260619 10:25:50.054262 10030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:50.054396 10030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [term 1 LEADER]: Becoming Leader. State: Replica: 52543dc79fc742d49708eb3ba81a08d3, State: Running, Role: LEADER
I20260619 10:25:50.054479 10030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.054872 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52543dc79fc742d49708eb3ba81a08d3. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.054970 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:50.055166 10052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:50.055372 10052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:50.057322 9984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [term 1 FOLLOWER]: Refusing update from remote peer 52543dc79fc742d49708eb3ba81a08d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:50.057600 10030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:50.058807 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52543dc79fc742d49708eb3ba81a08d3. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.058899 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.063951 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.053905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.19.65.189:42889)
0619 10:25:50.053949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.053951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.054696 (+ 745us) server_negotiation.cc:407] Connection header received
0619 10:25:50.054848 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.054849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.054877 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.054912 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.056464 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.057165 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.058667 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.058974 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.059028 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.060809 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.060810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.060811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.060819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.062546 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.062963 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.062964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.062964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.063022 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.063266 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.063267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.063267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.063434 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.063508 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.063764 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.063852 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:25:50.064178 9935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } attempt: 3
Jun 19 10:25:50 dist-test-slave-xd2q krb5kdc[9779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864749, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.65.189@KRBTEST.COM for kudu/127.19.65.188@KRBTEST.COM
I20260619 10:25:50.071158 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.054258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.19.65.188:38625)
0619 10:25:50.054316 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.054318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.059886 (+ 5568us) server_negotiation.cc:407] Connection header received
0619 10:25:50.060059 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.060060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.060083 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.060112 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.061542 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.062557 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.064396 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.064713 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.064754 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.067346 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.067347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.067347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.067354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.069411 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.070056 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.070057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.070058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.070112 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.070435 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.070436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.070436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.070555 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.070623 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.070992 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.071066 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:25:50.072464 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.053795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.19.65.190:33701)
0619 10:25:50.053845 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.053847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.059103 (+ 5256us) server_negotiation.cc:407] Connection header received
0619 10:25:50.059227 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.059228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.059239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.059274 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.063369 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.064111 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.065902 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.066187 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.066231 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.068639 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.068640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.068641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.068648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.070921 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.071458 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.071459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.071460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.071522 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.071840 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.071841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.071841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.071970 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.072078 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.072288 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.072371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:25:50.072597 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.072712 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:50.072759 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.072842 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.072885 10053 mvcc.cc:205] Tried to move back new op lower bound from 7298518016232042496 to 7298518016223862784. Current Snapshot: MvccSnapshot[applied={T|T < 7298518016232042496}]
I20260619 10:25:50.073047 10054 mvcc.cc:205] Tried to move back new op lower bound from 7298518016232042496 to 7298518016223862784. Current Snapshot: MvccSnapshot[applied={T|T < 7298518016232042496}]
I20260619 10:25:50.076205 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.064518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.65.188:38625 (local address 127.0.0.1:58822)
0619 10:25:50.064815 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:50.064826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:50.064873 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:50.065132 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:50.065133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:50.065296 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:50.065334 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:50.065335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.066286 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.066287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:50.067098 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:50.067101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.067644 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.067645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:50.067718 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.068227 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:50.068247 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:50.069901 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:50.074989 (+ 5088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.074990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.074990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.075206 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.075543 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:50.075544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:50.075544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:50.075626 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:50.075911 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:50.075912 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:50.075955 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:25:50.076019 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:25:50.076109 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260619 10:25:50.076273 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:50.064827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.19.65.188:38625)
0619 10:25:50.064977 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.064979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.064986 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:25:50.065006 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.065007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.065017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.065412 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.065457 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.066187 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.067208 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:50.067518 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:50.067626 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:50.073095 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:50.073097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:50.073097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:50.073105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:50.074900 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.075347 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.075348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.075348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.075418 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:50.075699 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:50.075700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:50.075700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:50.075834 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:50.075897 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:50.076069 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 10:25:50.076157 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260619 10:25:50.077548 10052 catalog_manager.cc:1347] Generated new cluster ID: dd500b7fca474cccaa7815d8ddd83cdb
I20260619 10:25:50.077597 10052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:50.078006 9935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: Bootstrap starting.
I20260619 10:25:50.078758 9935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:50.079097 9935 log.cc:824] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:50.079919 9935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: No bootstrap required, opened a new log
I20260619 10:25:50.081992 9935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.082150 9935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a0b32065d04dc3a39787b5564acc4c, State: Initialized, Role: FOLLOWER
I20260619 10:25:50.082242 9935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } }
I20260619 10:25:50.082582 9935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:50.084193 9907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [term 0 FOLLOWER]: Advancing to term 1
W20260619 10:25:50.084551 10071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:50.084605 10071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:50.085076 9907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [term 1 FOLLOWER]: Refusing update from remote peer 52543dc79fc742d49708eb3ba81a08d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:50.085367 10030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:50.085387 9909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52543dc79fc742d49708eb3ba81a08d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "10a0b32065d04dc3a39787b5564acc4c"
I20260619 10:25:50.086227 9909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52543dc79fc742d49708eb3ba81a08d3 in term 1.
I20260619 10:25:50.085386 10060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.086383 10060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.087769 10075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52543dc79fc742d49708eb3ba81a08d3. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.087860 10075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.087903 10075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.087967 10075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:25:50.089659 10052 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:50.090339 10052 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:50.095404 10073 mvcc.cc:205] Tried to move back new op lower bound from 7298518016232042496 to 7298518016223862784. Current Snapshot: MvccSnapshot[applied={T|T < 7298518016232042496}]
I20260619 10:25:50.099360 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52543dc79fc742d49708eb3ba81a08d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52543dc79fc742d49708eb3ba81a08d3" member_type: VOTER last_known_addr { host: "127.19.65.190" port: 33701 } } peers { permanent_uuid: "10a0b32065d04dc3a39787b5564acc4c" member_type: VOTER last_known_addr { host: "127.19.65.189" port: 42889 } } peers { permanent_uuid: "61b21736c6b141fc910071b21c42a7eb" member_type: VOTER last_known_addr { host: "127.19.65.188" port: 38625 } } }
I20260619 10:25:50.099429 10052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3: Generated new TSK 0
I20260619 10:25:50.099588 10052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:25:50.099462 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52543dc79fc742d49708eb3ba81a08d3 [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:50.122177 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.120836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.19.65.190:33701)
0619 10:25:50.120879 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.120882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.121494 (+ 612us) server_negotiation.cc:407] Connection header received
0619 10:25:50.121601 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.121602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.121613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.121644 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.122112 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46388: BlockingRecv error: recv got EOF from 127.0.0.1:46388 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:25:50.123191 10056 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.121184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.19.65.188:38625)
0619 10:25:50.121223 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.121225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.122525 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 10:25:50.122716 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.122718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.122728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.122757 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.123137 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58838: BlockingRecv error: recv got EOF from 127.0.0.1:58838 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:25:50.124317 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.123684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.19.65.189:42889)
0619 10:25:50.123731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.123733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.123833 (+ 100us) server_negotiation.cc:407] Connection header received
0619 10:25:50.123926 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.123927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.123937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.123966 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.124262 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54634: BlockingRecv error: recv got EOF from 127.0.0.1:54634 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 10:25:50.126619 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.125845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.19.65.189:42889)
0619 10:25:50.125878 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.125879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.126117 (+ 238us) server_negotiation.cc:407] Connection header received
0619 10:25:50.126223 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.126225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.126241 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.126271 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.126579 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54646: BlockingRecv error: recv got EOF from 127.0.0.1:54646 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:50.129611 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.128902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.19.65.189:42889)
0619 10:25:50.128934 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.128936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.129143 (+ 207us) server_negotiation.cc:407] Connection header received
0619 10:25:50.129236 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.129237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.129246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.129275 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.129571 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54660: BlockingRecv error: recv got EOF from 127.0.0.1:54660 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:50.133440 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.132728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.19.65.189:42889)
0619 10:25:50.132757 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.132759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.132952 (+ 193us) server_negotiation.cc:407] Connection header received
0619 10:25:50.133042 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.133043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.133052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.133079 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.133403 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54674: BlockingRecv error: recv got EOF from 127.0.0.1:54674 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 10:25:50.139389 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.138594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.19.65.189:42889)
0619 10:25:50.138625 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.138627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.138836 (+ 209us) server_negotiation.cc:407] Connection header received
0619 10:25:50.138929 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.138930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.138939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.138967 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.139266 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54690: BlockingRecv error: recv got EOF from 127.0.0.1:54690 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:25:50.149312 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.148617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.19.65.189:42889)
0619 10:25:50.148645 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.148647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.148847 (+ 200us) server_negotiation.cc:407] Connection header received
0619 10:25:50.148946 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.148947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.148955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.148980 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.149265 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54696: BlockingRecv error: recv got EOF from 127.0.0.1:54696 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 10:25:50.167351 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.166527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.19.65.189:42889)
0619 10:25:50.166563 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.166565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.166793 (+ 228us) server_negotiation.cc:407] Connection header received
0619 10:25:50.166904 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.166905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.166915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.166943 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.167271 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54710: BlockingRecv error: recv got EOF from 127.0.0.1:54710 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 10:25:50.201443 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.200701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.19.65.189:42889)
0619 10:25:50.200745 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.200747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.200898 (+ 151us) server_negotiation.cc:407] Connection header received
0619 10:25:50.200990 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.200991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.201003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.201032 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.201379 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54724: BlockingRecv error: recv got EOF from 127.0.0.1:54724 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 10:25:50.267398 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.266660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.19.65.189:42889)
0619 10:25:50.266710 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.266712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.266852 (+ 140us) server_negotiation.cc:407] Connection header received
0619 10:25:50.266887 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.266888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.266899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.266982 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.267332 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54732: BlockingRecv error: recv got EOF from 127.0.0.1:54732 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 10:25:50.397390 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.396604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.19.65.189:42889)
0619 10:25:50.396682 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.396685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.396802 (+ 117us) server_negotiation.cc:407] Connection header received
0619 10:25:50.396920 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.396921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.396933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.396992 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.397326 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54746: BlockingRecv error: recv got EOF from 127.0.0.1:54746 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 10:25:50.655130 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:25:50.654471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.19.65.189:42889)
0619 10:25:50.654543 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:50.654546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:50.654557 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:25:50.654655 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:50.654656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:50.654668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:50.654736 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:50.655064 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54760: BlockingRecv error: recv got EOF from 127.0.0.1:54760 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:25:51.013548 10027 catalog_manager.cc:1259] Loaded cluster ID: dd500b7fca474cccaa7815d8ddd83cdb
I20260619 10:25:51.013612 10027 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: loading cluster ID for follower catalog manager: success
I20260619 10:25:51.014360 10027 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: acquiring CA information for follower catalog manager: success
I20260619 10:25:51.014597 10027 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61b21736c6b141fc910071b21c42a7eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:25:51.084993 10071 catalog_manager.cc:1259] Loaded cluster ID: dd500b7fca474cccaa7815d8ddd83cdb
I20260619 10:25:51.085065 10071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: loading cluster ID for follower catalog manager: success
I20260619 10:25:51.085819 10071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: acquiring CA information for follower catalog manager: success
I20260619 10:25:51.086073 10071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10a0b32065d04dc3a39787b5564acc4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:25:51.170910 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:51.168206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.19.65.189:42889)
0619 10:25:51.168315 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:51.168318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:51.168408 (+ 90us) server_negotiation.cc:407] Connection header received
0619 10:25:51.168518 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:51.168519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:51.168580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:51.168662 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:51.168814 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.169318 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.170058 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.170283 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.170356 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:51.170537 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:51.170664 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:51.170744 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:51.170792 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 10:25:51.170812 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260619 10:25:51.174696 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:51.172576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.19.65.188:38625)
0619 10:25:51.172682 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:51.172684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:51.172690 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:25:51.172723 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:51.172724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:51.172753 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:51.172815 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:51.172892 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.173379 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.174052 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.174236 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.174283 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:51.174393 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:51.174485 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:51.174531 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:51.174565 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 10:25:51.174578 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260619 10:25:51.178954 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:51.176296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.19.65.190:33701)
0619 10:25:51.176400 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:51.176402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:51.176547 (+ 145us) server_negotiation.cc:407] Connection header received
0619 10:25:51.176593 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:51.176594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:51.176643 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:51.176727 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:51.176865 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.177389 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178146 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.178381 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178461 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:51.178628 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:51.178723 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:51.178784 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:51.178846 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:25:51.178869 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260619 10:25:51.179433 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:51.176452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.19.65.188:38625)
0619 10:25:51.176488 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:51.176489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:51.176532 (+ 43us) server_negotiation.cc:407] Connection header received
0619 10:25:51.176598 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:51.176599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:51.176608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:51.176653 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:51.176828 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.177536 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178537 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.178822 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178861 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:51.179044 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:51.179094 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:51.179139 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:51.179338 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 10:25:51.179360 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:25:51.179612 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:51.176239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.19.65.189:42889)
0619 10:25:51.176277 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:51.176279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:51.176359 (+ 80us) server_negotiation.cc:407] Connection header received
0619 10:25:51.176467 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:51.176468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:51.176480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:51.176511 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:51.176657 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.177423 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178561 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:51.178828 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:51.178886 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:51.179160 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:51.179202 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:51.179231 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:51.179533 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 10:25:51.179555 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:25:51.184047 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9792
I20260619 10:25:51.189189 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9867
I20260619 10:25:51.193853 19718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskadFLwT/build/release/bin/kudu with pid 9942
2026-06-19T10:25:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2757 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5506 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (261660 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST